Nov 03 10:26:04.447690 ip-10-0-134-12 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 10:26:04.447701 ip-10-0-134-12 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 10:26:04.447708 ip-10-0-134-12 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 10:26:04.447937 ip-10-0-134-12 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 10:26:14.449860 ip-10-0-134-12 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 10:26:14.449876 ip-10-0-134-12 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e37a38c8e62c4636853189a56b64e968 --
Nov 03 10:28:33.341082 ip-10-0-134-12 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 10:28:33.750280 ip-10-0-134-12 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:28:33.750280 ip-10-0-134-12 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 10:28:33.750280 ip-10-0-134-12 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:28:33.750280 ip-10-0-134-12 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:28:33.757348 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757259    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:33.757348 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757345    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:33.757348 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757349    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:33.757348 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757352    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:33.757348 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757355    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:33.757348 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757358    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757361    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757364    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757367    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757369    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757372    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757376    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757378    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757381    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757384    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757387    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757390    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757392    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757395    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757397    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757407    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757410    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757413    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757416    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757418    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:33.757569 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757421    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757423    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757426    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757430    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757434    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757437    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757440    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757443    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757446    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757449    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757452    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757455    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757458    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757461    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757465    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757469    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757472    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757475    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:33.758061 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757478    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757480    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757483    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757486    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757488    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757491    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757494    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757496    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757499    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757502    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757504    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757507    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757510    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757512    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757514    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757517    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757519    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757522    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757525    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757527    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:33.758479 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757530    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757532    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757535    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757538    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757540    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757543    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757545    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757549    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757552    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757555    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757558    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757560    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757563    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757565    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757568    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757570    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757573    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757575    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757578    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757581    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:33.758971 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757584    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757587    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757590    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757592    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757595    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757597    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757600    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757602    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757605    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757607    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757609    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757612    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757614    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757617    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757620    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757622    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757624    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757628    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757631    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757635    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:33.759443 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757638    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757640    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.757643    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758445    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758452    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758454    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758457    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758460    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758463    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758466    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758469    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758471    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758474    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758477    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758480    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758483    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758486    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758489    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758492    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:33.759922 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758495    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758506    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758508    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758511    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758513    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758516    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758520    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758525    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758528    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758531    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758534    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758537    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758540    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758543    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758545    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758548    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758550    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758553    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758556    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758558    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:33.760463 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758561    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758563    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758566    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758568    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758571    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758574    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758577    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758584    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758587    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758590    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758592    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758595    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758598    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758600    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758603    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758606    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758608    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758611    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758613    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758616    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:33.760953 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758618    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758621    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758623    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758625    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758628    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758631    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758633    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758636    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758638    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758641    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758645    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758649    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758652    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758654    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758657    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758659    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758662    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758665    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758668    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:33.761424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758670    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758673    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758676    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758679    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758681    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758684    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758686    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758689    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758691    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758694    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758696    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758699    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758701    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758704    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758706    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758709    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758711    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758713    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758734    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:33.761899 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758736    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758739    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758741    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758744    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758746    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758749    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758755    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758762    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758764    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758767    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758771    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.758773    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759283    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759288    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759292    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759295    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759297    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759300    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759303    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759306    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:33.762362 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759308    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759311    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759313    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759316    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759318    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759320    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759323    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759325    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759328    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759331    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759333    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759336    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759339    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759343    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759346    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759349    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759351    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759354    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:33.762847 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759357    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759361    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759364    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759366    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759369    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759371    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759374    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759376    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759379    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759381    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759384    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759386    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759389    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759391    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759394    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759396    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759399    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759401    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759403    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759406    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:33.763263 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759408    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759411    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759413    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759415    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759417    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759420    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759422    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759425    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759427    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759429    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759432    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759434    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759437    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759441    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759444    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759447    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759449    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759452    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759454    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:33.763789 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759457    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759460    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759463    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759467    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759469    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759472    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759474    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759477    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759479    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759482    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759484    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759487    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759490    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759492    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759495    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759497    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759499    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759502    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759504    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759507    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:33.764248 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759509    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759512    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759514    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759518    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759522    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759525    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759529    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759532    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759534    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759537    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759539    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759542    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759544    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759547    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759549    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759552    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759555    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759558    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759560    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:33.764770 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759563    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.759565    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760412    2573 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760422    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760428    2573 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760433    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760437    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760441    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760445    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760449    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760452    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760456    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760459    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760462    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760465    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760469    2573 flags.go:64] FLAG: --cgroup-root=""
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760472    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760475    2573 flags.go:64] FLAG: --client-ca-file=""
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760478    2573 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760481    2573 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760486    2573 flags.go:64] FLAG: --cluster-domain=""
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760490    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760493    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760496    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 10:28:33.765196 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760500    2573 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760509    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760512    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760515    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760518    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760521    2573 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760524    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760527    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760530    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760533    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760537    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760540    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760543    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760546    2573 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760549    2573 flags.go:64] FLAG: --enable-server="true"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760552    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760556    2573 flags.go:64] FLAG: --event-burst="100"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760559    2573 flags.go:64] FLAG: --event-qps="50"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760562    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760564    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760567    2573 flags.go:64] FLAG: --eviction-hard=""
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760571    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760574    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760579    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760582    2573 flags.go:64] FLAG: --eviction-soft=""
Nov 03 10:28:33.765807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760585    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760588    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760591    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760594    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760597    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760599    2573 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760602    2573 flags.go:64] FLAG: --feature-gates=""
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760606    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760609    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760612    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760615    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760618    2573 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760621    2573 flags.go:64] FLAG: --help="false"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760624    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760629    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760632    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760635    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760638    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760642    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760645    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760647    2573 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760650    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760653    2573 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 10:28:33.766399 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760656    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760659    2573 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760662    2573 flags.go:64] FLAG: --kube-reserved=""
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760665    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760667    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760670    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760673    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760677    2573 flags.go:64] FLAG: --lock-file=""
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760680    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760683    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760686    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760692    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760695    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760698    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760700    2573 flags.go:64] FLAG: --logging-format="text"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760703    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760706    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760709    2573 flags.go:64] FLAG: --manifest-url=""
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760712    2573 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760731    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760734    2573 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760738    2573 flags.go:64] FLAG: --max-pods="110"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760741    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760744    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760749    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 10:28:33.767150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760752    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760755    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760759    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760761    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760768    2573 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760771    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760774    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760777    2573 flags.go:64] FLAG: --pod-cidr=""
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760780    2573 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760783    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760786    2573 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760789    2573 flags.go:64] FLAG: --port="10250"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760792    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760795    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d381cf900aeee9ec"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760799    2573 flags.go:64] FLAG: --qos-reserved=""
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760802    2573 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760805    2573 flags.go:64] FLAG: --register-node="true"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760808    2573 flags.go:64] FLAG: --register-with-taints=""
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760813    2573 flags.go:64] FLAG: --registry-burst="10"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760816    2573 flags.go:64] FLAG: --registry-qps="5"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760819    2573 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760822    2573 flags.go:64] FLAG: --reserved-memory=""
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760826    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760829    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760832    2573 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760835    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 10:28:33.768227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760838    2573 flags.go:64] FLAG: --runonce="false"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760841    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760844    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760847    2573 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760850    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760853    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760857    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760860    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760863    2573 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760866    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760869    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760872    2573 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760875    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760878    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760881    2573 flags.go:64] FLAG: --system-cgroups=""
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760883    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760889    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760892    2573 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760895    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760900    2573 flags.go:64] FLAG: --tls-min-version=""
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760903    2573 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760906    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760909    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760912    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760916    2573 flags.go:64] FLAG: --v="2"
Nov 03 10:28:33.769383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760921    2573 flags.go:64] FLAG: --version="false"
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760925    2573 flags.go:64] FLAG: --vmodule=""
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760929    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.760932    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761073    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761077    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761080    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761083    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761086    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761088    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761093    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761097    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761099    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761102    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761105    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761108    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761111    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761113    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761116    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:33.770491 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761119    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761122    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761124    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761127    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761129    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761132    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761134    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761137    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761140    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761143    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761145    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761148    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761152    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761154    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761157    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761159    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761161    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761164    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761167    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:33.771359 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761169    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761171    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761174    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761177    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761179    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761181    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761184    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761186    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761189    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761192    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761195    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761197    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761200    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761202    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761205    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761207    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761210    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761212    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761214    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761217    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:33.771904 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761219    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761223    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761226    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761228    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761231    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761234    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761237    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761239    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761242    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761244    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761247    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761249    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761251    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761254    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761256    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761259    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761261    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761264    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761266    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761269    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:33.772697 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761271    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761274    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761277    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761279    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761282    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761284    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761287    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761289    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761292    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761294    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761297    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761300    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761304    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761308    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761311    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761313    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761316    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761320    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761322    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761325    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:33.773342 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761327    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761329    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761332    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761334    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761337    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761339    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761342    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761344    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761346    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761349    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761351    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761354    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.761405    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761443    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761448    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761457    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:33.773841 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761460    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761462    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761465    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761468    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761471    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761473    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761476    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761478    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761481    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761483    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761486    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761489    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761491    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761497    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761500    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761503    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761506    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761508    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761511    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761513    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:33.774328 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761516    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761518    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761520    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761523    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761525    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761528    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761530    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761532    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761535    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761537    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761539    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761542    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761545    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761547    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761549    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761552    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761554    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761557    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761559    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:33.774986 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761562    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761566    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761570    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761573    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761575    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761578    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761581    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761584    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761586    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761589    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761592    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761595    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761597    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761600    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761602    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761604    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761607    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761609    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:33.775596 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761612    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761614    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761616    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761619    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761621    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761623    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761626    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761628    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761631    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761634    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761636    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761639    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761641    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761644    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761646    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761648    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761651    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761653    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761655    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761658    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:33.776045 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761662    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761664    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761666    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761669    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761671    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761674    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761676    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761678    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761681    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761683    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761686    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761688    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761691    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761693    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761696    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761698    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761701    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761703    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761705    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761708    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:33.776520 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761710    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761729    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761732    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761735    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761737    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.761740    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.769308    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.769329    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.769352    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.769359    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769445    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769454    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769459    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769463    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769468    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769472    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769476    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769481    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769485    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769490    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:33.777082 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769494    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769498    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769502    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769507    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769511    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769516    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769520    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769524    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769529    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769533    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769537    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769542    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769545    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769550    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769554    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769559    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769563    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769567    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769572    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:33.777626 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769578    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769583    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769587    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769591    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769595    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769599    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769604    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769608    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769613    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769617    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769622    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769626    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769630    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769634    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769638    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769643    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769647    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769652    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769656    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769660    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:33.778093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769664    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769668    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769673    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769677    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769681    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769686    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769691    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769695    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769699    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769703    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769708    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769712    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769736    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769746    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769752    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769757    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769761    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769766    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769770    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:33.778961 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769774    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769778    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769782    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769787    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769790    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769794    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769799    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769803    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769807    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769812    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769816    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769820    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769824    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769829    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769833    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769837    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769842    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769847    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769851    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:33.779403 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769855    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769862    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769868    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769873    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769878    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769882    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769887    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769891    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769896    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769900    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769904    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769908    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769912    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769916    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769921    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769925    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769929    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769933    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:33.779875 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.769937    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.769981    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770212    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770219    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770226    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770231    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770236    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770241    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770245    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770249    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770254    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770258    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770262    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770266    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770270    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770274    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:33.780315 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770279    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770283    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770287    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770292    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770296    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770300    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770304    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770309    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770316    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770324    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770329    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770333    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770337    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770341    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770345    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770350    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770353    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770357    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770361    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:33.780737 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770365    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770369    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770373    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770377    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770382    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770386    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770390    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770394    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770398    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770402    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770407    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770411    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770415    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770419    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770424    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770428    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770432    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770436    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770441    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770445    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:33.781178 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770449    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770454    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770459    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770463    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770467    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770471    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770475    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770479    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770483    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770487    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770491    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770495    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770506    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770510    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770514    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770518    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770522    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770526    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770530    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770535    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:33.781632 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770539    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770543    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770549    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770553    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770557    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770561    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770566    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770570    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770574    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770578    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770582    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770586    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770590    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770594    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770599    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770603    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770608    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770612    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770616    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:33.782130 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770620    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770624    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770628    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770632    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770636    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770640    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770644    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770648    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770652    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770656    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770660    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770664    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770668    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770673    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.770711    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:28:33.782579 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770790    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770797    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770802    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770809    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770813    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770818    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770822    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770826    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770830    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770834    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770839    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770843    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770847    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770852    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770856    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770861    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770865    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770869    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770873    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770878    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:28:33.783016 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770882    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770887    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770891    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770895    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770899    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770904    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770908    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770912    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770916    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770920    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770925    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770929    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770933    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770937    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770941    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770947    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770951    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770955    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770959    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770963    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:28:33.783787 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770968    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770972    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770976    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770980    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770984    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770989    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770994    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.770998    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771003    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771007    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771011    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771015    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771019    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771023    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771027    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771032    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771036    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771040    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771044    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:28:33.784424 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771048    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771053    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771057    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771063    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771067    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771071    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771076    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771080    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771086    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771091    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771095    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771099    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771104    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771108    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771112    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771116    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771121    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771125    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771129    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:28:33.784913 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771133    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771138    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771143    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771150    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771156    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771161    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771165    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771169    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771173    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771177    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771181    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771186    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771190    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771194    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771199    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771203    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771210    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771215    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771219    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771223    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:28:33.785386 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771227    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771233    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771237    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771244    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771250    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771254    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771259    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:33.771265    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.771349    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.775135    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.776523    2573 server.go:1014] "Starting client certificate rotation"
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.776616    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.776648    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.778825    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:28:33.785868 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.780234    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:28:33.795426 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.795404    2573 log.go:25] "Validated CRI v1 runtime API"
Nov 03 10:28:33.799190 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.799168    2573 log.go:25] "Validated CRI v1 image API"
Nov 03 10:28:33.800049 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.800018    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 10:28:33.804407 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.804376    2573 fs.go:135] Filesystem UUIDs: map[546f2653-f94d-4dc3-9c48-08cebc0ad93a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d44f2494-a1b2-4252-8156-f0b821cbbd13:/dev/nvme0n1p3]
Nov 03 10:28:33.804505 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.804405    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 10:28:33.804569 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.804559    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 10:28:33.809586 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.809464    2573 manager.go:217] Machine: {Timestamp:2026-11-03 10:28:33.808194459 +0000 UTC m=+0.340285357 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100423 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26ce5bc24ffbf2c5dde0968cfa8d13 SystemUUID:ec26ce5b-c24f-fbf2-c5dd-e0968cfa8d13 BootID:e37a38c8-e62c-4636-8531-89a56b64e968 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c3:8b:74:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c3:8b:74:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:b6:b9:7e:87:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 10:28:33.809586 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.809577    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 10:28:33.809757 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.809685    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 10:28:33.812878 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.812856    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 10:28:33.813011 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.812879    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-12.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 10:28:33.813058 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.813013    2573 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 10:28:33.813058 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.813021    2573 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 10:28:33.813107 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.813067    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 10:28:33.813107 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.813080    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 10:28:33.813867 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.813857    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 10:28:33.817689 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.817677    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 10:28:33.817819 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.817809    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 10:28:33.817851 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.817845    2573 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 10:28:33.817876 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.817857    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 10:28:33.817876 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.817867    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 10:28:33.817876 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.817877    2573 kubelet.go:405] "Adding apiserver pod source"
Nov 03 10:28:33.817979 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.817885    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 10:28:33.818279 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.818254    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 10:28:33.821097 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.821072    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 10:28:33.821500 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.821485    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsddh"
Nov 03 10:28:33.827475 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.827459    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 10:28:33.828728 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.828678    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsddh"
Nov 03 10:28:33.828880 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.828865    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 10:28:33.828921 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.828899    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 10:28:33.829056 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829047    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 10:28:33.829089 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829063    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 10:28:33.829089 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829069    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 10:28:33.829089 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829074    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 10:28:33.829089 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829079    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 10:28:33.829089 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829085    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 10:28:33.829938 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829926    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 10:28:33.829938 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829938    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 10:28:33.830016 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829946    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 10:28:33.830016 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829952    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 10:28:33.830016 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829961    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 10:28:33.830016 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.829970    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 10:28:33.831214 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.831204    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 10:28:33.831245 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.831215    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 10:28:33.836356 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.836338    2573 server.go:1263] "Started kubelet"
Nov 03 10:28:33.836473 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.836438    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 10:28:33.836561 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.836477    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 10:28:33.836561 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.836535    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 10:28:33.837200 ip-10-0-134-12 systemd[1]: Started Kubernetes Kubelet.
Nov 03 10:28:33.839278 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.839259    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 10:28:33.840184 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.840169    2573 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 10:28:33.842083 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.842063    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:33.842387 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.842371    2573 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-134-12.us-east-2.compute.internal" not found
Nov 03 10:28:33.843671 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.843655    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:33.844639 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.844620    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 10:28:33.844895 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:33.844880    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 10:28:33.845569 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.845557    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 10:28:33.845680 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:33.845656    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-12.us-east-2.compute.internal\" not found"
Nov 03 10:28:33.846177 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846158    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 10:28:33.846177 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846178    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 10:28:33.846295 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846160    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 10:28:33.846328 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846309    2573 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 10:28:33.846328 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846319    2573 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 10:28:33.846488 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846473    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 10:28:33.846527 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846492    2573 factory.go:55] Registering systemd factory
Nov 03 10:28:33.846527 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846508    2573 factory.go:223] Registration of the systemd container factory successfully
Nov 03 10:28:33.846820 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846803    2573 factory.go:153] Registering CRI-O factory
Nov 03 10:28:33.846875 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846824    2573 factory.go:223] Registration of the crio container factory successfully
Nov 03 10:28:33.846875 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846844    2573 factory.go:103] Registering Raw factory
Nov 03 10:28:33.846875 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.846859    2573 manager.go:1199] Started watching for new ooms in manager
Nov 03 10:28:33.847809 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.847795    2573 manager.go:319] Starting recovery of all containers
Nov 03 10:28:33.852563 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.852539    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:33.853555 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:33.853528    2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-12.us-east-2.compute.internal\" not found" node="ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:33.854694 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.854667    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 10:28:33.857919 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.857884    2573 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-134-12.us-east-2.compute.internal" not found
Nov 03 10:28:33.859571 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.859555    2573 manager.go:324] Recovery completed
Nov 03 10:28:33.864840 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.864827    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:33.866670 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.866655    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:33.866770 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.866686    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:33.866770 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.866696    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:33.867260 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.867248    2573 cpu_manager.go:226] "Starting" policy="none"
Nov 03 10:28:33.867260 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.867259    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 10:28:33.867368 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.867275    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 10:28:33.867368 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.867249    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:33.868023 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.868006    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:33.868085 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.868035    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:33.868085 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.868047    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:33.869516 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.869502    2573 policy_none.go:50] "Start"
Nov 03 10:28:33.869571 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.869520    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 10:28:33.869571 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.869530    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 10:28:33.871491 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.871481    2573 policy_none.go:44] "Start"
Nov 03 10:28:33.912678 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.912533    2573 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-134-12.us-east-2.compute.internal" not found
Nov 03 10:28:33.919309 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.919270    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 10:28:33.921811 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.921795    2573 manager.go:341] "Starting Device Plugin manager"
Nov 03 10:28:33.921897 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:33.921841    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 10:28:33.921897 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.921853    2573 server.go:87] "Starting device plugin registration server"
Nov 03 10:28:33.922102 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.922090    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 10:28:33.922147 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.922102    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 10:28:33.922231 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.922204    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 10:28:33.922331 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.922313    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 10:28:33.922448 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:33.922327    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 10:28:33.922922 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:33.922902    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 10:28:33.923000 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:33.922953    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-12.us-east-2.compute.internal\" not found"
Nov 03 10:28:34.011261 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.011197    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 10:28:34.011261 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.011226    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 10:28:34.011261 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.011249    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 10:28:34.013186 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.011282    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 10:28:34.016829 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.016806    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:34.022895 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.022876    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:28:34.023797 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.023777    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:28:34.023874 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.023816    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:28:34.023874 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.023831    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:28:34.023874 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.023868    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.035269 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.035254    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.111595 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.111554    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"]
Nov 03 10:28:34.113895 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.113871    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.115701 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.115683    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.147078 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.147056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3041328657ebd4fdb2a0f070aa3e29b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"3041328657ebd4fdb2a0f070aa3e29b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.147192 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.147085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.147192 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.147113    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.148041 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.148027    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.150978 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.150961    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.158702 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.158687    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:28:34.228349 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.228321    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:28:34.247622 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.247588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.247756 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.247634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.247756 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.247667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3041328657ebd4fdb2a0f070aa3e29b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"3041328657ebd4fdb2a0f070aa3e29b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.247756 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.247679    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.247895 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.247761    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3041328657ebd4fdb2a0f070aa3e29b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"3041328657ebd4fdb2a0f070aa3e29b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.247895 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.247765    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.462409 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.462330    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.532067 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.532036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:28:34.775701 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.775624    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 10:28:34.776180 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.775798    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 10:28:34.776180 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.775800    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 10:28:34.776180 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.775801    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 10:28:34.819030 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.819006    2573 apiserver.go:52] "Watching apiserver"
Nov 03 10:28:34.825666 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.825642    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 10:28:34.825945 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.825929    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-vgs7g","openshift-multus/multus-additional-cni-plugins-nb997","openshift-multus/network-metrics-daemon-wm4x2","openshift-network-operator/iptables-alerter-9ft6m","kube-system/global-pull-secret-syncer-w4txd","openshift-image-registry/node-ca-67pqq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-cd6vm","kube-system/konnectivity-agent-9d2nl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9","openshift-multus/multus-j7kpq","openshift-dns/node-resolver-crz98","openshift-ovn-kubernetes/ovnkube-node-6t57h","kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"]
Nov 03 10:28:34.828254 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.828237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.830532 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.830494    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.831969 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.831953    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:28:34.832166 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.832145    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 10:28:34.832212 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.832182    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-trm8g\""
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.832745    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.832834    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.833425    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 10:23:34 +0000 UTC" deadline="2028-06-10 01:51:26.918992025 +0000 UTC"
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.833508    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14031h22m52.085502189s"
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.833794    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.833836    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.833924    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.833966    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.834105    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wbpkj\""
Nov 03 10:28:34.835376 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.834451    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 10:28:34.835985 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.835971    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.838011 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.837990    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.838131 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.838109    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:34.840053 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.840035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.842071 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.842055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:34.842164 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.842133    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:34.844150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.844136    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:34.845662 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.845647    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 10:28:34.846378 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.846339    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.848655 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.848638    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850105 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-device-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.850184 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850118    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn96l\" (UniqueName: \"kubernetes.io/projected/17db0bdd-c610-416d-b20e-b0f51a839205-kube-api-access-bn96l\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.850184 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcfs\" (UniqueName: \"kubernetes.io/projected/e0ecce6a-ad31-4a68-9298-5f5618ece509-kube-api-access-zhcfs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850184 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.850277 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysconfig\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.850277 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.850336 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850280    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-socket-dir-parent\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850336 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-dbus\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.850393 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qrlf\" (UniqueName: \"kubernetes.io/projected/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-kube-api-access-9qrlf\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:34.850393 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-netns\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850450 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-kubelet\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850450 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-conf-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850514 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.850514 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88501d15-b241-4af1-af7a-eabc4c417a08-konnectivity-ca\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:34.850573 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-cnibin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850573 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-bin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850630 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850570    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-multus\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850630 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-hostroot\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850685 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-daemon-config\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.850685 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850675    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-conf\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.850767 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.850767 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850738    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.850824 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850772    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fffd3590-b7f0-41c9-b24f-1575b39c8f25-iptables-alerter-script\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.850824 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850778    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:34.850824 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850802    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fvz\" (UniqueName: \"kubernetes.io/projected/fffd3590-b7f0-41c9-b24f-1575b39c8f25-kube-api-access-82fvz\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.850928 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-sys\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.850985 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.850933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-var-lib-kubelet\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851031 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-kubernetes\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851107 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-run\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851166 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851128    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-sys-fs\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.851166 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.851263 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851230    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:34.851263 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851259    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-etc-kubernetes\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.851361 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-system-cni-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.851361 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851310    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48f7q\" (UniqueName: \"kubernetes.io/projected/00141100-9f3c-4e38-b3d5-04a31b4c76e3-kube-api-access-48f7q\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.851361 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851343    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-etc-tuned\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851499 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851377    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-kubelet-config\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.851499 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851425    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-tmp\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851499 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851467    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkggv\" (UniqueName: \"kubernetes.io/projected/2a1f00d4-439a-46b1-98fe-b595af207890-kube-api-access-tkggv\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851587 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851503    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:34.851587 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-registration-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.851587 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-system-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.851587 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-cni-binary-copy\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.851706 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-host\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.851706 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851706 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fffd3590-b7f0-41c9-b24f-1575b39c8f25-host-slash\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.851706 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zvn\" (UniqueName: \"kubernetes.io/projected/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-kube-api-access-89zvn\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.851706 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88501d15-b241-4af1-af7a-eabc4c417a08-agent-certs\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:34.851706 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851708    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-serviceca\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.851929 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851744    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-socket-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.851929 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851792    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.851929 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cnibin\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.851929 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-modprobe-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851929 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851892    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-systemd\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.851929 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851913    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-k8s-cni-cncf-io\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.852138 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851939    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.852138 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.851981    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-lib-modules\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.852138 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.852009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-multus-certs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.852138 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.852046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-host\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.853132 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.853117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.857996 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.857977    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 10:28:34.878044 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.878010    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hwqvg"
Nov 03 10:28:34.885780 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.885761    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hwqvg"
Nov 03 10:28:34.932370 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.932346    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 10:28:34.932502 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.932483    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 10:28:34.932636 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.932620    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 10:28:34.932689 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.932621    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 10:28:34.932689 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.932675    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zcr5r\""
Nov 03 10:28:34.932793 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.932701    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nxhw\""
Nov 03 10:28:34.933776 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.933757    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 10:28:34.933776 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.933759    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 10:28:34.933776 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.933786    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 10:28:34.933776 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.933764    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.933801    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.933762    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xjpbn\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.933763    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.933998    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.934004    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.934009    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.934017    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.934063    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.934085    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.934087    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 10:28:34.934141 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.934109    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 10:28:34.935553 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.935533    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4c7q2\""
Nov 03 10:28:34.935755 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.935711    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-khtn4\""
Nov 03 10:28:34.935899 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.935878    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jkrmv\""
Nov 03 10:28:34.935965 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.935903    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5hs5j\""
Nov 03 10:28:34.936051 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.935577    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 10:28:34.936270 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.936250    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 10:28:34.946834 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.946820    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 10:28:34.952428 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952410    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-etc-tuned\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.952492 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-kubelet-config\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.952492 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51410712-fd08-4429-8d93-c0266660f335-tmp-dir\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:34.952492 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-etc-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.952492 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952486    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.952679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952508    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-bin\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.952679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952537    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-kubelet-config\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.952679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952546    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-config\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.952679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-tmp\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.952679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkggv\" (UniqueName: \"kubernetes.io/projected/2a1f00d4-439a-46b1-98fe-b595af207890-kube-api-access-tkggv\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.952944 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952732    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:34.952944 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-registration-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.952944 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-system-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.952944 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952828    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 10:28:34.952944 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-cni-binary-copy\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.952944 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952881    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-registration-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.952944 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-host\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952924    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-system-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.952995    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-host\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953011    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fffd3590-b7f0-41c9-b24f-1575b39c8f25-host-slash\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953032    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zvn\" (UniqueName: \"kubernetes.io/projected/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-kube-api-access-89zvn\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-ovn\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953062    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88501d15-b241-4af1-af7a-eabc4c417a08-agent-certs\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-serviceca\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953107    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-systemd-units\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953109    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953128    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-systemd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-socket-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953197    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cnibin\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.953229 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953219    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-netns\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-modprobe-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953263    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fffd3590-b7f0-41c9-b24f-1575b39c8f25-host-slash\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-systemd\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-k8s-cni-cncf-io\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953316    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-systemd\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953316    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-cni-binary-copy\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953379    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-k8s-cni-cncf-io\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-kubelet\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-lib-modules\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-multus-certs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-var-lib-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953504    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cnibin\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-host\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-device-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.953887 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn96l\" (UniqueName: \"kubernetes.io/projected/17db0bdd-c610-416d-b20e-b0f51a839205-kube-api-access-bn96l\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953625    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcfs\" (UniqueName: \"kubernetes.io/projected/e0ecce6a-ad31-4a68-9298-5f5618ece509-kube-api-access-zhcfs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953457    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-socket-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953703    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-serviceca\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953739    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953775    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-multus-certs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953806    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-host\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953602    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-modprobe-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovn-node-metrics-cert\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953894    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-lib-modules\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.953937    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-device-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysconfig\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-socket-dir-parent\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954120    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-dbus\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-node-log\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.954679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954163    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysconfig\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954168    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qrlf\" (UniqueName: \"kubernetes.io/projected/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-kube-api-access-9qrlf\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954201    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-netns\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-kubelet\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-conf-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954267    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-socket-dir-parent\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-slash\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954312    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954318    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-log-socket\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88501d15-b241-4af1-af7a-eabc4c417a08-konnectivity-ca\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954359    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954365    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-kubelet\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-cnibin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954390    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-conf-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-bin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.955554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-multus\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954445    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-dbus\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.954448    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954361    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-netns\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-hostroot\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.954538    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret podName:430e1cec-2732-4742-8eed-d2ba0fc4caf3 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:35.454519075 +0000 UTC m=+1.986609965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret") pod "global-pull-secret-syncer-w4txd" (UID: "430e1cec-2732-4742-8eed-d2ba0fc4caf3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954587    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-bin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954623    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-multus\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-daemon-config\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954672    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-hostroot\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954741    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954755    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-cnibin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-conf\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fffd3590-b7f0-41c9-b24f-1575b39c8f25-iptables-alerter-script\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fvz\" (UniqueName: \"kubernetes.io/projected/fffd3590-b7f0-41c9-b24f-1575b39c8f25-kube-api-access-82fvz\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.956135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954930    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-conf\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954894    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88501d15-b241-4af1-af7a-eabc4c417a08-konnectivity-ca\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.954951    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51410712-fd08-4429-8d93-c0266660f335-hosts-file\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955000    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f26td\" (UniqueName: \"kubernetes.io/projected/51410712-fd08-4429-8d93-c0266660f335-kube-api-access-f26td\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-sys\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955070    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-var-lib-kubelet\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-netd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-env-overrides\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955166    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-daemon-config\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbnh\" (UniqueName: \"kubernetes.io/projected/71f6bbd9-acd8-4534-8adb-ca7cd139d383-kube-api-access-jqbnh\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-kubernetes\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-run\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-sys-fs\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955329    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955348    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-run\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.956613 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-etc-kubernetes\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955308    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955380    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-sys\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955440    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-sys-fs\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955432    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-var-lib-kubelet\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955487    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-etc-kubernetes\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955489    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-kubernetes\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955508    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-system-cni-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955550    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48f7q\" (UniqueName: \"kubernetes.io/projected/00141100-9f3c-4e38-b3d5-04a31b4c76e3-kube-api-access-48f7q\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.955557    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955583    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-system-cni-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-script-lib\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.955617    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fffd3590-b7f0-41c9-b24f-1575b39c8f25-iptables-alerter-script\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.955639    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs podName:64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b nodeName:}" failed. No retries permitted until 2026-11-03 10:28:35.455628893 +0000 UTC m=+1.987719792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs") pod "network-metrics-daemon-wm4x2" (UID: "64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.956551    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-tmp\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.956651    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-etc-tuned\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.957185 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.956865    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88501d15-b241-4af1-af7a-eabc4c417a08-agent-certs\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:34.958689 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.958668    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:34.958689 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.958685    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:34.958853 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.958695    2573 projected.go:196] Error preparing data for projected volume kube-api-access-fh24b for pod openshift-network-diagnostics/network-check-target-cd6vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:34.958853 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:34.958750    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b podName:cbcaff87-f298-42b7-8fce-0850c567d9c7 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:35.458740056 +0000 UTC m=+1.990830942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fh24b" (UniqueName: "kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b") pod "network-check-target-cd6vm" (UID: "cbcaff87-f298-42b7-8fce-0850c567d9c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:34.961404 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.961383    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkggv\" (UniqueName: \"kubernetes.io/projected/2a1f00d4-439a-46b1-98fe-b595af207890-kube-api-access-tkggv\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:34.961793 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.961521    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn96l\" (UniqueName: \"kubernetes.io/projected/17db0bdd-c610-416d-b20e-b0f51a839205-kube-api-access-bn96l\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:34.961793 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.961688    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcfs\" (UniqueName: \"kubernetes.io/projected/e0ecce6a-ad31-4a68-9298-5f5618ece509-kube-api-access-zhcfs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:34.962300 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.962281    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zvn\" (UniqueName: \"kubernetes.io/projected/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-kube-api-access-89zvn\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:34.962734 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.962697    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qrlf\" (UniqueName: \"kubernetes.io/projected/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-kube-api-access-9qrlf\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:34.965363 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.965338    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fvz\" (UniqueName: \"kubernetes.io/projected/fffd3590-b7f0-41c9-b24f-1575b39c8f25-kube-api-access-82fvz\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:34.966094 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:34.966077    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48f7q\" (UniqueName: \"kubernetes.io/projected/00141100-9f3c-4e38-b3d5-04a31b4c76e3-kube-api-access-48f7q\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:35.056409 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51410712-fd08-4429-8d93-c0266660f335-tmp-dir\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:35.056409 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-etc-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056409 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-bin\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-config\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056460    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056475    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-bin\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-ovn\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056510    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-ovn\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056462    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-etc-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056531    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-systemd-units\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-systemd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-netns\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056601    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-systemd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056576    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-systemd-units\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056624    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-kubelet\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056634    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-netns\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.056645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056648    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-var-lib-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056673    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-kubelet\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056699    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-var-lib-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056738    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51410712-fd08-4429-8d93-c0266660f335-tmp-dir\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056755    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovn-node-metrics-cert\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-node-log\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056738    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-slash\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-log-socket\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056867    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-node-log\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056902    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-slash\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056913    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-log-socket\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056913    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51410712-fd08-4429-8d93-c0266660f335-hosts-file\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f26td\" (UniqueName: \"kubernetes.io/projected/51410712-fd08-4429-8d93-c0266660f335-kube-api-access-f26td\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:35.057307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.056998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-netd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057817 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.057016    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-env-overrides\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057817 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.057041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbnh\" (UniqueName: \"kubernetes.io/projected/71f6bbd9-acd8-4534-8adb-ca7cd139d383-kube-api-access-jqbnh\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057817 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.057052    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-netd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057817 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.057060    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-config\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057817 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.057057    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51410712-fd08-4429-8d93-c0266660f335-hosts-file\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:35.057817 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.057091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-script-lib\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057817 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.057362    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-env-overrides\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.057817 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.057542    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-script-lib\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.059310 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.059288    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovn-node-metrics-cert\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.064318 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.064294    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbnh\" (UniqueName: \"kubernetes.io/projected/71f6bbd9-acd8-4534-8adb-ca7cd139d383-kube-api-access-jqbnh\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.064963 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.064945    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f26td\" (UniqueName: \"kubernetes.io/projected/51410712-fd08-4429-8d93-c0266660f335-kube-api-access-f26td\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:35.163283 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.163254    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:28:35.182210 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.182184    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:28:35.197054 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.197030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:28:35.203090 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.203072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:28:35.205232 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.205212    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:35.209224 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.209199    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf179e0cd67151dd9aa7b9a87412922fb.slice/crio-91a30875268765bed1cc83c1bdc52cf8b049b9032f79a710d0ad0eeceeb86c5e WatchSource:0}: Error finding container 91a30875268765bed1cc83c1bdc52cf8b049b9032f79a710d0ad0eeceeb86c5e: Status 404 returned error can't find the container with id 91a30875268765bed1cc83c1bdc52cf8b049b9032f79a710d0ad0eeceeb86c5e
Nov 03 10:28:35.209440 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.209425    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3041328657ebd4fdb2a0f070aa3e29b6.slice/crio-fee12141e68d64c054e7de05847e6cb79216771ec54a0695882cef68ff72781b WatchSource:0}: Error finding container fee12141e68d64c054e7de05847e6cb79216771ec54a0695882cef68ff72781b: Status 404 returned error can't find the container with id fee12141e68d64c054e7de05847e6cb79216771ec54a0695882cef68ff72781b
Nov 03 10:28:35.209612 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.209594    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00141100_9f3c_4e38_b3d5_04a31b4c76e3.slice/crio-136a60959a855bfbeff212a5c5e618b6968c28f5b662533fc576518fe14e4e38 WatchSource:0}: Error finding container 136a60959a855bfbeff212a5c5e618b6968c28f5b662533fc576518fe14e4e38: Status 404 returned error can't find the container with id 136a60959a855bfbeff212a5c5e618b6968c28f5b662533fc576518fe14e4e38
Nov 03 10:28:35.210782 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.210765    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffd3590_b7f0_41c9_b24f_1575b39c8f25.slice/crio-161058e45d6663ee796adea96ae77c39d8bec7a3153b59883d19651862dc197c WatchSource:0}: Error finding container 161058e45d6663ee796adea96ae77c39d8bec7a3153b59883d19651862dc197c: Status 404 returned error can't find the container with id 161058e45d6663ee796adea96ae77c39d8bec7a3153b59883d19651862dc197c
Nov 03 10:28:35.216106 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.216087    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:28:35.219093 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.219074    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88501d15_b241_4af1_af7a_eabc4c417a08.slice/crio-d8420a329aa507a59b761e8778e0c3ae8c04c20ea63ccc74d9328547862d9685 WatchSource:0}: Error finding container d8420a329aa507a59b761e8778e0c3ae8c04c20ea63ccc74d9328547862d9685: Status 404 returned error can't find the container with id d8420a329aa507a59b761e8778e0c3ae8c04c20ea63ccc74d9328547862d9685
Nov 03 10:28:35.220151 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.220130    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7532f5ee_e2ed_40a2_9d9e_277a392e1c9f.slice/crio-dffa86620a6045e2442c2d70fc23b67884611501d81e8bc9760245dbd795e15e WatchSource:0}: Error finding container dffa86620a6045e2442c2d70fc23b67884611501d81e8bc9760245dbd795e15e: Status 404 returned error can't find the container with id dffa86620a6045e2442c2d70fc23b67884611501d81e8bc9760245dbd795e15e
Nov 03 10:28:35.231689 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.231670    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:28:35.240581 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.240561    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17db0bdd_c610_416d_b20e_b0f51a839205.slice/crio-a2f8a0a98ec816ccc44f3d6ed26a5bd85ecc09db14a1226215d142e04c236257 WatchSource:0}: Error finding container a2f8a0a98ec816ccc44f3d6ed26a5bd85ecc09db14a1226215d142e04c236257: Status 404 returned error can't find the container with id a2f8a0a98ec816ccc44f3d6ed26a5bd85ecc09db14a1226215d142e04c236257
Nov 03 10:28:35.245667 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.245649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j7kpq"
Nov 03 10:28:35.254846 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.254825    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ecce6a_ad31_4a68_9298_5f5618ece509.slice/crio-8115cbe0c0262d974bb9566912e76cc706913703b7a4bd2194018a99a7fb0b64 WatchSource:0}: Error finding container 8115cbe0c0262d974bb9566912e76cc706913703b7a4bd2194018a99a7fb0b64: Status 404 returned error can't find the container with id 8115cbe0c0262d974bb9566912e76cc706913703b7a4bd2194018a99a7fb0b64
Nov 03 10:28:35.276577 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.276551    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-crz98"
Nov 03 10:28:35.285934 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.285912    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51410712_fd08_4429_8d93_c0266660f335.slice/crio-53f0ab87129c8ba04c72fa95466ee09074e903d5635d7eaf2aa0840957ce8578 WatchSource:0}: Error finding container 53f0ab87129c8ba04c72fa95466ee09074e903d5635d7eaf2aa0840957ce8578: Status 404 returned error can't find the container with id 53f0ab87129c8ba04c72fa95466ee09074e903d5635d7eaf2aa0840957ce8578
Nov 03 10:28:35.294901 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.294884    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:35.304067 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:28:35.304047    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f6bbd9_acd8_4534_8adb_ca7cd139d383.slice/crio-31eb62fa6614e04ced7dc4a8147116968a2b7343fa2d787f6c815248b9136fa1 WatchSource:0}: Error finding container 31eb62fa6614e04ced7dc4a8147116968a2b7343fa2d787f6c815248b9136fa1: Status 404 returned error can't find the container with id 31eb62fa6614e04ced7dc4a8147116968a2b7343fa2d787f6c815248b9136fa1
Nov 03 10:28:35.460030 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.459995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:35.460030 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.460036    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:35.460226 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.460060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:35.460226 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:35.460149    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:35.460226 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:35.460161    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:35.460226 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:35.460179    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:35.460226 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:35.460194    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:35.460226 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:35.460205    2573 projected.go:196] Error preparing data for projected volume kube-api-access-fh24b for pod openshift-network-diagnostics/network-check-target-cd6vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:35.460226 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:35.460209    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret podName:430e1cec-2732-4742-8eed-d2ba0fc4caf3 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:36.4601939 +0000 UTC m=+2.992284790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret") pod "global-pull-secret-syncer-w4txd" (UID: "430e1cec-2732-4742-8eed-d2ba0fc4caf3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:35.460226 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:35.460226    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs podName:64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b nodeName:}" failed. No retries permitted until 2026-11-03 10:28:36.460219211 +0000 UTC m=+2.992310095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs") pod "network-metrics-daemon-wm4x2" (UID: "64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:35.460488 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:35.460237    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b podName:cbcaff87-f298-42b7-8fce-0850c567d9c7 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:36.460231486 +0000 UTC m=+2.992322371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fh24b" (UniqueName: "kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b") pod "network-check-target-cd6vm" (UID: "cbcaff87-f298-42b7-8fce-0850c567d9c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:35.582859 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.582787    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:35.593979 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:35.593955    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:37.012602 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.012461    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:36 +0000 UTC" deadline="2028-05-30 03:36:11.549901637 +0000 UTC"
Nov 03 10:28:37.012602 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.012556    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13769h7m34.537349371s"
Nov 03 10:28:37.137284 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.137254    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:37.137436 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.137391    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:37.157807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.157735    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vgs7g" event={"ID":"2a1f00d4-439a-46b1-98fe-b595af207890","Type":"ContainerStarted","Data":"07a6654d333e1f34daca2e18fc3a1e675817ed26cf1ac6220a217c6c4c9d6c98"}
Nov 03 10:28:37.165049 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.165010    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-crz98" event={"ID":"51410712-fd08-4429-8d93-c0266660f335","Type":"ContainerStarted","Data":"53f0ab87129c8ba04c72fa95466ee09074e903d5635d7eaf2aa0840957ce8578"}
Nov 03 10:28:37.169400 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.169367    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-67pqq" event={"ID":"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f","Type":"ContainerStarted","Data":"dffa86620a6045e2442c2d70fc23b67884611501d81e8bc9760245dbd795e15e"}
Nov 03 10:28:37.172258 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.172229    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"31eb62fa6614e04ced7dc4a8147116968a2b7343fa2d787f6c815248b9136fa1"}
Nov 03 10:28:37.175162 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.175130    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j7kpq" event={"ID":"e0ecce6a-ad31-4a68-9298-5f5618ece509","Type":"ContainerStarted","Data":"8115cbe0c0262d974bb9566912e76cc706913703b7a4bd2194018a99a7fb0b64"}
Nov 03 10:28:37.186313 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.186280    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9" event={"ID":"17db0bdd-c610-416d-b20e-b0f51a839205","Type":"ContainerStarted","Data":"a2f8a0a98ec816ccc44f3d6ed26a5bd85ecc09db14a1226215d142e04c236257"}
Nov 03 10:28:37.195363 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.195333    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9d2nl" event={"ID":"88501d15-b241-4af1-af7a-eabc4c417a08","Type":"ContainerStarted","Data":"d8420a329aa507a59b761e8778e0c3ae8c04c20ea63ccc74d9328547862d9685"}
Nov 03 10:28:37.198684 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.198651    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9ft6m" event={"ID":"fffd3590-b7f0-41c9-b24f-1575b39c8f25","Type":"ContainerStarted","Data":"161058e45d6663ee796adea96ae77c39d8bec7a3153b59883d19651862dc197c"}
Nov 03 10:28:37.212551 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.212517    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerStarted","Data":"136a60959a855bfbeff212a5c5e618b6968c28f5b662533fc576518fe14e4e38"}
Nov 03 10:28:37.223077 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.223005    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"f179e0cd67151dd9aa7b9a87412922fb","Type":"ContainerStarted","Data":"91a30875268765bed1cc83c1bdc52cf8b049b9032f79a710d0ad0eeceeb86c5e"}
Nov 03 10:28:37.227950 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.227068    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"3041328657ebd4fdb2a0f070aa3e29b6","Type":"ContainerStarted","Data":"fee12141e68d64c054e7de05847e6cb79216771ec54a0695882cef68ff72781b"}
Nov 03 10:28:37.306002 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.305912    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.591142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.591206    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:37.591257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.591377    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.591440    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret podName:430e1cec-2732-4742-8eed-d2ba0fc4caf3 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:39.591422353 +0000 UTC m=+5.000314319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret") pod "global-pull-secret-syncer-w4txd" (UID: "430e1cec-2732-4742-8eed-d2ba0fc4caf3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.591507    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.591538    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs podName:64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b nodeName:}" failed. No retries permitted until 2026-11-03 10:28:39.591527928 +0000 UTC m=+5.000419897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs") pod "network-metrics-daemon-wm4x2" (UID: "64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.591615    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.591631    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.591643    2573 projected.go:196] Error preparing data for projected volume kube-api-access-fh24b for pod openshift-network-diagnostics/network-check-target-cd6vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:37.592067 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:37.591679    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b podName:cbcaff87-f298-42b7-8fce-0850c567d9c7 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:39.59166806 +0000 UTC m=+5.000560026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fh24b" (UniqueName: "kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b") pod "network-check-target-cd6vm" (UID: "cbcaff87-f298-42b7-8fce-0850c567d9c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:38.013383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:38.013333    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:36 +0000 UTC" deadline="2028-04-30 06:28:05.953493069 +0000 UTC"
Nov 03 10:28:38.013383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:38.013375    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13051h59m27.940122814s"
Nov 03 10:28:38.135182 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:38.135123    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:38.135365 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:38.135303    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:38.135767 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:38.135745    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:38.135880 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:38.135847    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:39.137682 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:39.137649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:39.138223 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.137799    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:39.613073 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:39.613032    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:39.613237 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:39.613112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:39.613237 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:39.613151    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:39.613237 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.613210    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:39.613441 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.613284    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret podName:430e1cec-2732-4742-8eed-d2ba0fc4caf3 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:43.613264356 +0000 UTC m=+9.022156334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret") pod "global-pull-secret-syncer-w4txd" (UID: "430e1cec-2732-4742-8eed-d2ba0fc4caf3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:39.613441 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.613296    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:39.613441 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.613316    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:39.613441 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.613331    2573 projected.go:196] Error preparing data for projected volume kube-api-access-fh24b for pod openshift-network-diagnostics/network-check-target-cd6vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:39.613441 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.613378    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b podName:cbcaff87-f298-42b7-8fce-0850c567d9c7 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:43.613363698 +0000 UTC m=+9.022255661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fh24b" (UniqueName: "kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b") pod "network-check-target-cd6vm" (UID: "cbcaff87-f298-42b7-8fce-0850c567d9c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:39.613710 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.613459    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:39.613710 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:39.613494    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs podName:64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b nodeName:}" failed. No retries permitted until 2026-11-03 10:28:43.613483041 +0000 UTC m=+9.022375008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs") pod "network-metrics-daemon-wm4x2" (UID: "64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:40.135211 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:40.135182    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:40.135365 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:40.135304    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:40.135606 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:40.135588    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:40.135674 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:40.135663    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:40.256974 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:40.256560    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerStarted","Data":"b811939b1a45d8544dfdff6c58def701438fedce6ba63eace3a2e01669ce3ed5"}
Nov 03 10:28:41.137911 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:41.137849    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:41.138036 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:41.138011    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:41.272143 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:41.272081    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9d2nl" event={"ID":"88501d15-b241-4af1-af7a-eabc4c417a08","Type":"ContainerStarted","Data":"3a6ca2adb056500d14666a03a1e6747d24b9c7778facdddc93eb01850a31f23e"}
Nov 03 10:28:41.294011 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:41.291820    2573 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="b811939b1a45d8544dfdff6c58def701438fedce6ba63eace3a2e01669ce3ed5" exitCode=0
Nov 03 10:28:41.294011 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:41.291910    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerDied","Data":"b811939b1a45d8544dfdff6c58def701438fedce6ba63eace3a2e01669ce3ed5"}
Nov 03 10:28:41.301231 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:41.300858    2573 generic.go:356] "Generic (PLEG): container finished" podID="f179e0cd67151dd9aa7b9a87412922fb" containerID="cb2da0cf3383e14be726d9d44071fb9bc9ec6299465cfe5c7449158ba2f6afb2" exitCode=0
Nov 03 10:28:41.301231 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:41.300903    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"f179e0cd67151dd9aa7b9a87412922fb","Type":"ContainerDied","Data":"cb2da0cf3383e14be726d9d44071fb9bc9ec6299465cfe5c7449158ba2f6afb2"}
Nov 03 10:28:41.310826 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:41.310768    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9d2nl" podStartSLOduration=1.7034140880000002 podStartE2EDuration="6.310747464s" totalImagesPullingTime="4.607333376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:41.287212125 +0000 UTC m=+6.696104112" watchObservedRunningTime="2026-11-03 10:28:41.310747464 +0000 UTC m=+6.719639449"
Nov 03 10:28:42.136020 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:42.134981    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:42.136020 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:42.135141    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:42.136020 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:42.135549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:42.136020 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:42.135644    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:42.309433 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:42.308655    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"f179e0cd67151dd9aa7b9a87412922fb","Type":"ContainerStarted","Data":"f2a166e97a08fbe0238171c622cd915060b31331d3a730db48a62b9a97a94aef"}
Nov 03 10:28:42.312319 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:42.311887    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"3041328657ebd4fdb2a0f070aa3e29b6","Type":"ContainerStarted","Data":"2ce1d4a71baf36d12295b12d6be36d7fd9b005fb2de2e785de986f7c006a624a"}
Nov 03 10:28:42.316171 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:42.316037    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-67pqq" event={"ID":"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f","Type":"ContainerStarted","Data":"f16f70a16383dfe382e6c620241e6a6498bf5bf0502894959bff15a5b88389bb"}
Nov 03 10:28:42.324167 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:42.323401    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal" podStartSLOduration=7.323386442 podStartE2EDuration="7.323386442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:42.322626798 +0000 UTC m=+7.731518783" watchObservedRunningTime="2026-11-03 10:28:42.323386442 +0000 UTC m=+7.732278427"
Nov 03 10:28:42.336259 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:42.335441    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal" podStartSLOduration=7.335422782 podStartE2EDuration="7.335422782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:42.334809473 +0000 UTC m=+7.743701460" watchObservedRunningTime="2026-11-03 10:28:42.335422782 +0000 UTC m=+7.744314770"
Nov 03 10:28:43.135374 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:43.135343    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:43.135557 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.135491    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:43.322570 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:43.321749    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9" event={"ID":"17db0bdd-c610-416d-b20e-b0f51a839205","Type":"ContainerStarted","Data":"bf0fdf2428175478cf45e4bfe3f8502f1d7f079ce3ee1ebd9bd7f1491c27a964"}
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:43.649482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:43.649556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:43.649599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.649735    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.649797    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs podName:64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b nodeName:}" failed. No retries permitted until 2026-11-03 10:28:51.649779291 +0000 UTC m=+17.058671256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs") pod "network-metrics-daemon-wm4x2" (UID: "64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.650068    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.650134    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret podName:430e1cec-2732-4742-8eed-d2ba0fc4caf3 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:51.650116904 +0000 UTC m=+17.059008880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret") pod "global-pull-secret-syncer-w4txd" (UID: "430e1cec-2732-4742-8eed-d2ba0fc4caf3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.650072    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.650163    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.650177    2573 projected.go:196] Error preparing data for projected volume kube-api-access-fh24b for pod openshift-network-diagnostics/network-check-target-cd6vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:43.650259 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:43.650232    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b podName:cbcaff87-f298-42b7-8fce-0850c567d9c7 nodeName:}" failed. No retries permitted until 2026-11-03 10:28:51.650217557 +0000 UTC m=+17.059109522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fh24b" (UniqueName: "kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b") pod "network-check-target-cd6vm" (UID: "cbcaff87-f298-42b7-8fce-0850c567d9c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:44.135202 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:44.135165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:44.135370 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:44.135300    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:44.136021 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:44.135709    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:44.136021 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:44.135810    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:44.183561 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:44.183527    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:44.184559 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:44.184541    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:44.199169 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:44.199113    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-67pqq" podStartSLOduration=3.691590368 podStartE2EDuration="9.199094719s" totalImagesPullingTime="5.507504351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:42.348969507 +0000 UTC m=+7.757861497" watchObservedRunningTime="2026-11-03 10:28:44.199094719 +0000 UTC m=+9.607986709"
Nov 03 10:28:44.744413 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:44.744373    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 10:28:45.054170 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:45.054069    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T10:28:44.744395309Z","UUID":"3d4d48c4-2741-4a29-aabf-0ac24d01598c","Handler":null,"Name":"","Endpoint":""}
Nov 03 10:28:45.056496 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:45.056221    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 10:28:45.056496 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:45.056270    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 10:28:45.136088 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:45.136056    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:45.136248 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:45.136199    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:45.328838 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:45.328720    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j7kpq" event={"ID":"e0ecce6a-ad31-4a68-9298-5f5618ece509","Type":"ContainerStarted","Data":"57b85e0ba8562c6ccd9d12a3da2e21ddcf8510d3b01928c54728a9b94d886e4e"}
Nov 03 10:28:45.331811 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:45.331779    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9" event={"ID":"17db0bdd-c610-416d-b20e-b0f51a839205","Type":"ContainerStarted","Data":"6e62d3e894f1bd78de0a5262e31ffe4a04f86ba9adab284d6ce3cc94f54bfd5b"}
Nov 03 10:28:45.334472 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:45.334446    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-crz98" event={"ID":"51410712-fd08-4429-8d93-c0266660f335","Type":"ContainerStarted","Data":"ecaeba5ed8d8775f3ddddea337ca5c9402783cfdb1ba83ab0144865f6ff72fcb"}
Nov 03 10:28:45.347828 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:45.347748    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-j7kpq" podStartSLOduration=1.916431977 podStartE2EDuration="10.347732629s" totalImagesPullingTime="8.431300652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:45.347450618 +0000 UTC m=+10.756342602" watchObservedRunningTime="2026-11-03 10:28:45.347732629 +0000 UTC m=+10.756624615"
Nov 03 10:28:46.134991 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:46.134958    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:46.135458 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:46.134958    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:46.135458 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:46.135069    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:46.135458 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:46.135173    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:46.340314 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:46.340238    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9" event={"ID":"17db0bdd-c610-416d-b20e-b0f51a839205","Type":"ContainerStarted","Data":"134a69dd362067b364742375ca711c9ec410a097da9c0c15082949415c604847"}
Nov 03 10:28:46.343159 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:46.342546    2573 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="d38a2ab9a83266c7548888e0773d1f3ee5d29375755784e14e7570e940e119ac" exitCode=0
Nov 03 10:28:46.343159 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:46.342649    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerDied","Data":"d38a2ab9a83266c7548888e0773d1f3ee5d29375755784e14e7570e940e119ac"}
Nov 03 10:28:46.356833 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:46.356778    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9" podStartSLOduration=2.16237577 podStartE2EDuration="11.356762101s" totalImagesPullingTime="9.194386331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:46.356496361 +0000 UTC m=+11.765388345" watchObservedRunningTime="2026-11-03 10:28:46.356762101 +0000 UTC m=+11.765654086"
Nov 03 10:28:46.357110 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:46.357059    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-crz98" podStartSLOduration=2.964164501 podStartE2EDuration="11.357050725s" totalImagesPullingTime="8.392886224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:45.366210653 +0000 UTC m=+10.775102641" watchObservedRunningTime="2026-11-03 10:28:46.357050725 +0000 UTC m=+11.765942710"
Nov 03 10:28:47.135594 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:47.135570    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:47.136052 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:47.135685    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:47.347561 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:47.347295    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9ft6m" event={"ID":"fffd3590-b7f0-41c9-b24f-1575b39c8f25","Type":"ContainerStarted","Data":"c20860e14bb515bf5c0f125d2ea8af6f43fbcfa8edf187b2e5ccd212e5e3d9d4"}
Nov 03 10:28:47.349464 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:47.349437    2573 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="7cd48c855586ef5d29bd0e1f73010966da654d2e74e014e34473f3d6eb8e12d9" exitCode=0
Nov 03 10:28:47.349614 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:47.349511    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerDied","Data":"7cd48c855586ef5d29bd0e1f73010966da654d2e74e014e34473f3d6eb8e12d9"}
Nov 03 10:28:47.351873 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:47.351845    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vgs7g" event={"ID":"2a1f00d4-439a-46b1-98fe-b595af207890","Type":"ContainerStarted","Data":"0925596175d537766386a027082486173294974ccf581fdfe7170d46ab800d30"}
Nov 03 10:28:47.362507 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:47.362463    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9ft6m" podStartSLOduration=3.662504358 podStartE2EDuration="12.362450243s" totalImagesPullingTime="8.699945885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:47.362061158 +0000 UTC m=+12.770953143" watchObservedRunningTime="2026-11-03 10:28:47.362450243 +0000 UTC m=+12.771342227"
Nov 03 10:28:48.134935 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:48.134885    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:48.134935 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:48.134908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:48.135190 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:48.135030    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:48.135190 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:48.135142    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:48.518934 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:48.518898    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:48.519400 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:48.519101    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 10:28:48.519546 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:48.519529    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:28:48.532819 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:48.532777    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vgs7g" podStartSLOduration=3.051195744 podStartE2EDuration="13.532764163s" totalImagesPullingTime="10.481568419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:47.394423936 +0000 UTC m=+12.803315919" watchObservedRunningTime="2026-11-03 10:28:48.532764163 +0000 UTC m=+13.941656147"
Nov 03 10:28:49.134989 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:49.134954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:49.135160 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:49.135102    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:49.357715 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:49.357683    2573 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="0d4c25b46974f6020bea1a91a84b4305c6ce106d5c818357dc7b650be5347c5b" exitCode=0
Nov 03 10:28:49.357897 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:49.357778    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerDied","Data":"0d4c25b46974f6020bea1a91a84b4305c6ce106d5c818357dc7b650be5347c5b"}
Nov 03 10:28:50.135684 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:50.135649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:50.136340 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:50.135696    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:50.136340 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:50.135795    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:50.136340 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:50.135950    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:51.135034 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:51.134993    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:51.135219 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.135149    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:51.714876 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:51.714845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:51.714893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:51.714938    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.715022    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.715025    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.715049    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.715073    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs podName:64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b nodeName:}" failed. No retries permitted until 2026-11-03 10:29:07.715060524 +0000 UTC m=+33.123952500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs") pod "network-metrics-daemon-wm4x2" (UID: "64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.715074    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.715089    2573 projected.go:196] Error preparing data for projected volume kube-api-access-fh24b for pod openshift-network-diagnostics/network-check-target-cd6vm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.715088    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret podName:430e1cec-2732-4742-8eed-d2ba0fc4caf3 nodeName:}" failed. No retries permitted until 2026-11-03 10:29:07.715079405 +0000 UTC m=+33.123971366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret") pod "global-pull-secret-syncer-w4txd" (UID: "430e1cec-2732-4742-8eed-d2ba0fc4caf3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 10:28:51.715289 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:51.715151    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b podName:cbcaff87-f298-42b7-8fce-0850c567d9c7 nodeName:}" failed. No retries permitted until 2026-11-03 10:29:07.715131473 +0000 UTC m=+33.124023446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fh24b" (UniqueName: "kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b") pod "network-check-target-cd6vm" (UID: "cbcaff87-f298-42b7-8fce-0850c567d9c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 10:28:52.135699 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.135538    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:52.135793 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.135604    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:52.135793 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:52.135772    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:52.135899 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:52.135875    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:52.366985 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.366955    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:28:52.367349 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.367322    2573 generic.go:356] "Generic (PLEG): container finished" podID="71f6bbd9-acd8-4534-8adb-ca7cd139d383" containerID="137fae92e957983988ff501237fd5241b2af6a217a9fbad09542d09308618d26" exitCode=1
Nov 03 10:28:52.367472 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.367396    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"a3864709d26b4124f2835b47413f273c260b040b690eeb245bb5cd03846a9b15"}
Nov 03 10:28:52.367532 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.367483    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"ea5659b910f62a255e0ec295244ba709e2df3ff8ceec44310c0c046f276758e0"}
Nov 03 10:28:52.367587 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.367534    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"90dabd26cf4caabae3f011c84b8f88377640b0d2f4ca494bcb8b57bc1994e051"}
Nov 03 10:28:52.367640 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.367582    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"3d1dc5f4dc07cdcc77c73b2d227a60be7dd5aaf25d9b72f27d9646bda8242305"}
Nov 03 10:28:52.367640 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.367632    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerDied","Data":"137fae92e957983988ff501237fd5241b2af6a217a9fbad09542d09308618d26"}
Nov 03 10:28:52.367739 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.367682    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"a82ed16d193a41f3210644d19dff0bfa6dc32fc7d30515ff881d94dcf4cd0244"}
Nov 03 10:28:52.370439 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.370414    2573 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="70715e5516c3f005d69670851492704b14e33d98d8691289fbf75d9310944386" exitCode=0
Nov 03 10:28:52.370554 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:52.370457    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerDied","Data":"70715e5516c3f005d69670851492704b14e33d98d8691289fbf75d9310944386"}
Nov 03 10:28:53.135162 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:53.134989    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:53.135582 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:53.135244    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:53.375473 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:53.375440    2573 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="6765bb4cfc8f203cd18dc22572a18026eb8c7156b2d2ddd686acb57ebb168a52" exitCode=0
Nov 03 10:28:53.375625 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:53.375487    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerDied","Data":"6765bb4cfc8f203cd18dc22572a18026eb8c7156b2d2ddd686acb57ebb168a52"}
Nov 03 10:28:54.134645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:54.134610    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:54.134645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:54.134651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:54.134849 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:54.134729    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:54.134884 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:54.134842    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:54.384334 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:54.384310    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:28:54.384803 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:54.384728    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"88d368a5fdb4d1c2057cf960d82541edc0380ec5511929a6e14900af57e9016e"}
Nov 03 10:28:54.388296 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:54.388272    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb997" event={"ID":"00141100-9f3c-4e38-b3d5-04a31b4c76e3","Type":"ContainerStarted","Data":"aebf98fdfdcb7b82c8b33e834360d544ba96ad15932e2b9184b30f74e31a78ea"}
Nov 03 10:28:54.409935 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:54.409877    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nb997" podStartSLOduration=8.245180277 podStartE2EDuration="19.409864147s" totalImagesPullingTime="11.16468387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:54.409053781 +0000 UTC m=+19.817945765" watchObservedRunningTime="2026-11-03 10:28:54.409864147 +0000 UTC m=+19.818756130"
Nov 03 10:28:55.135446 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:55.135384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:55.135662 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:55.135521    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:56.134889 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:56.134854    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:56.135307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:56.134892    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:56.135307 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:56.134975    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:56.135307 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:56.135093    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:57.135059 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:57.135025    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:57.135750 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:57.135149    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:57.397841 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:57.397766    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:28:57.398150 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:57.398129    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"17680abee3e2befefcc630ace612642aca7484950fc93d33c07035dda5dd2094"}
Nov 03 10:28:57.398426 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:57.398410    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:57.398516 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:57.398481    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:57.398559 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:57.398526    2573 scope.go:122] "RemoveContainer" containerID="137fae92e957983988ff501237fd5241b2af6a217a9fbad09542d09308618d26"
Nov 03 10:28:57.415391 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:57.415372    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:58.135093 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:58.135048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:58.135539 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:58.135156    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:58.135539 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:58.135215    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:58.135539 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:58.135306    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:58.404230 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:58.404157    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:28:58.404488 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:58.404465    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" event={"ID":"71f6bbd9-acd8-4534-8adb-ca7cd139d383","Type":"ContainerStarted","Data":"106b495dcced20dd236ea2512cf5d304853789e64bec07119ba2c5e2144b5de2"}
Nov 03 10:28:58.404747 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:58.404729    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:58.420270 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:58.420243    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:28:58.430870 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:58.430828    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h" podStartSLOduration=8.08320915 podStartE2EDuration="23.430816154s" totalImagesPullingTime="15.347607004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:28:58.429430641 +0000 UTC m=+23.838322624" watchObservedRunningTime="2026-11-03 10:28:58.430816154 +0000 UTC m=+23.839708138"
Nov 03 10:28:59.135432 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:59.135204    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:59.135838 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:59.135487    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:28:59.167866 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:59.167836    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w4txd"]
Nov 03 10:28:59.168060 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:59.167990    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:28:59.168135 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:59.168111    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:28:59.171172 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:59.171147    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wm4x2"]
Nov 03 10:28:59.171996 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:59.171973    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cd6vm"]
Nov 03 10:28:59.172113 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:59.172068    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:28:59.172193 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:59.172165    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:28:59.406612 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:28:59.406540    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:28:59.406745 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:28:59.406676    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:29:01.135435 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:01.135403    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:01.135760 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:01.135475    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:01.135760 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:01.135557    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:01.135760 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:29:01.135575    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:29:01.135760 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:29:01.135644    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:29:01.135760 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:29:01.135708    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:29:03.135695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:03.135608    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:03.135695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:03.135610    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:03.136224 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:29:03.135709    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd6vm" podUID="cbcaff87-f298-42b7-8fce-0850c567d9c7"
Nov 03 10:29:03.136224 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:03.135736    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:03.136224 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:29:03.135821    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wm4x2" podUID="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b"
Nov 03 10:29:03.136224 ip-10-0-134-12 kubenswrapper[2573]: E1103 10:29:03.135881    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4txd" podUID="430e1cec-2732-4742-8eed-d2ba0fc4caf3"
Nov 03 10:29:04.373025 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.372998    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeReady"
Nov 03 10:29:04.373407 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.373130    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 10:29:04.413043 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.413008    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9zqgx"]
Nov 03 10:29:04.416075 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.416055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.418655 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.418633    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kjzdq"]
Nov 03 10:29:04.420252 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.420230    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 10:29:04.420441 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.420423    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w9j8\""
Nov 03 10:29:04.423224 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.423201    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 10:29:04.424303 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.424283    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:04.426177 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.426157    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9zqgx"]
Nov 03 10:29:04.433230 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.433212    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kjzdq"]
Nov 03 10:29:04.437792 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.437774    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-98krd"]
Nov 03 10:29:04.440467 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.440454    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.450749 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.450696    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-98krd"]
Nov 03 10:29:04.506175 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.506144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlmh\" (UniqueName: \"kubernetes.io/projected/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-kube-api-access-xhlmh\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.506285 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.506200    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-tmp-dir\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.506350 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.506328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6tf\" (UniqueName: \"kubernetes.io/projected/93cc76bb-77ed-493e-9207-c5bb8e30bc25-kube-api-access-4n6tf\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:04.506424 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.506387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-config-volume\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.506463 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.506434    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93cc76bb-77ed-493e-9207-c5bb8e30bc25-cert\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:04.506463 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.506457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-metrics-tls\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.519725 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.519706    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 10:29:04.519977 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.519958    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 10:29:04.520084 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.520066    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 10:29:04.520194 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.520116    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 10:29:04.520194 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.520130    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w9nrb\""
Nov 03 10:29:04.520194 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.520149    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 10:29:04.520194 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.520156    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 10:29:04.520194 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.520150    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 10:29:04.520194 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.520186    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5cc62\""
Nov 03 10:29:04.607356 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e913f7-bc17-4001-bed8-348466e5966c-data-volume\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.607356 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-config-volume\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.607580 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607385    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e913f7-bc17-4001-bed8-348466e5966c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.607580 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607484    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93cc76bb-77ed-493e-9207-c5bb8e30bc25-cert\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:04.607580 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-metrics-tls\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.607580 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlmh\" (UniqueName: \"kubernetes.io/projected/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-kube-api-access-xhlmh\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.607580 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6r65\" (UniqueName: \"kubernetes.io/projected/59e913f7-bc17-4001-bed8-348466e5966c-kube-api-access-g6r65\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.607580 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607573    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-tmp-dir\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.607845 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e913f7-bc17-4001-bed8-348466e5966c-crio-socket\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.607845 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e913f7-bc17-4001-bed8-348466e5966c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.607845 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607800    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6tf\" (UniqueName: \"kubernetes.io/projected/93cc76bb-77ed-493e-9207-c5bb8e30bc25-kube-api-access-4n6tf\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:04.607991 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.607906    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-tmp-dir\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.608088 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.608069    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-config-volume\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.611172 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.611147    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-metrics-tls\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.611417 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.611398    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93cc76bb-77ed-493e-9207-c5bb8e30bc25-cert\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:04.614826 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.614804    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlmh\" (UniqueName: \"kubernetes.io/projected/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-kube-api-access-xhlmh\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.615271 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.615250    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6tf\" (UniqueName: \"kubernetes.io/projected/93cc76bb-77ed-493e-9207-c5bb8e30bc25-kube-api-access-4n6tf\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:04.708814 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.708742    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6r65\" (UniqueName: \"kubernetes.io/projected/59e913f7-bc17-4001-bed8-348466e5966c-kube-api-access-g6r65\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.708814 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.708787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e913f7-bc17-4001-bed8-348466e5966c-crio-socket\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.708814 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.708810    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e913f7-bc17-4001-bed8-348466e5966c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.709099 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.709002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e913f7-bc17-4001-bed8-348466e5966c-data-volume\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.709099 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.709027    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e913f7-bc17-4001-bed8-348466e5966c-crio-socket\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.709099 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.709040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e913f7-bc17-4001-bed8-348466e5966c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.709340 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.709317    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e913f7-bc17-4001-bed8-348466e5966c-data-volume\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.709516 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.709500    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e913f7-bc17-4001-bed8-348466e5966c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.711271 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.711254    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e913f7-bc17-4001-bed8-348466e5966c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.716247 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.716223    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6r65\" (UniqueName: \"kubernetes.io/projected/59e913f7-bc17-4001-bed8-348466e5966c-kube-api-access-g6r65\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:04.734050 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.734031    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:04.738152 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.738132    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:04.755380 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:04.755347    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:05.030514 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.030440    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kjzdq"]
Nov 03 10:29:05.031070 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.031046    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9zqgx"]
Nov 03 10:29:05.035032 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:29:05.035009    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cc76bb_77ed_493e_9207_c5bb8e30bc25.slice/crio-313968a59c031db374af80688503703d2d2d03ee89dfe0f05701b168f404de52 WatchSource:0}: Error finding container 313968a59c031db374af80688503703d2d2d03ee89dfe0f05701b168f404de52: Status 404 returned error can't find the container with id 313968a59c031db374af80688503703d2d2d03ee89dfe0f05701b168f404de52
Nov 03 10:29:05.035599 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:29:05.035581    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cf9d6c_4dc2_49e0_93c9_d6115eeeb905.slice/crio-3e70903a7e12534ad0b15d17570f2bea210477a91ac151b60bdd2cec61dc5286 WatchSource:0}: Error finding container 3e70903a7e12534ad0b15d17570f2bea210477a91ac151b60bdd2cec61dc5286: Status 404 returned error can't find the container with id 3e70903a7e12534ad0b15d17570f2bea210477a91ac151b60bdd2cec61dc5286
Nov 03 10:29:05.039449 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.039431    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-98krd"]
Nov 03 10:29:05.041543 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:29:05.041524    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e913f7_bc17_4001_bed8_348466e5966c.slice/crio-1997dc5450e3cb7965eb3b05b42a223f7e84a2776fa84949036a56a23de384a2 WatchSource:0}: Error finding container 1997dc5450e3cb7965eb3b05b42a223f7e84a2776fa84949036a56a23de384a2: Status 404 returned error can't find the container with id 1997dc5450e3cb7965eb3b05b42a223f7e84a2776fa84949036a56a23de384a2
Nov 03 10:29:05.140981 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.137907    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:05.140981 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.138420    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:05.140981 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.140432    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:05.171308 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.171281    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-78k5n"]
Nov 03 10:29:05.190375 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.190343    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.209565 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.209535    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 10:29:05.209757 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.209743    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 10:29:05.209821 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.209759    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjjb4\""
Nov 03 10:29:05.210118 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.210101    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pthxs\""
Nov 03 10:29:05.210118 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.210107    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lq8vs\""
Nov 03 10:29:05.210243 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.210164    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 10:29:05.210645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.210629    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 10:29:05.211063 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.211047    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ssvsh\""
Nov 03 10:29:05.220859 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.220838    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 10:29:05.221005 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.220992    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 10:29:05.221593 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.221581    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 10:29:05.221818 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.221804    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 10:29:05.222161 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.222146    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 10:29:05.222226 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.222204    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 10:29:05.307838 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.307759    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-576b7767c8-btzbj"]
Nov 03 10:29:05.315982 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.315958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-sys\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.316064 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.315990    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.316064 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.316013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-wtmp\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.316201 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.316182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-textfile\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.316280 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.316264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-metrics-client-ca\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.316404 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.316387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-tls\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.316451 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.316413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8hh\" (UniqueName: \"kubernetes.io/projected/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-kube-api-access-bg8hh\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.316508 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.316481    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.316562 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.316545    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-root\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.331908 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.331883    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-576b7767c8-btzbj"]
Nov 03 10:29:05.332258 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.332242    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.338340 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.338320    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 10:29:05.338543 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.338489    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 10:29:05.338605 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.338558    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qpnvs\""
Nov 03 10:29:05.417198 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.417198 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417197    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqsg\" (UniqueName: \"kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417218    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-tls\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8hh\" (UniqueName: \"kubernetes.io/projected/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-kube-api-access-bg8hh\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417373    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-root\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.417695 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-sys\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417706    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-root\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417762    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-wtmp\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417788    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-sys\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417808    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-textfile\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417930    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-wtmp\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418130 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.417956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-metrics-client-ca\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418474 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.418286    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418474 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.418392    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-textfile\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.418562 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.418484    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-metrics-client-ca\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.420113 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.420083    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-tls\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.420214 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.420111    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.420375 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.420355    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 10:29:05.420548 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.420519    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 10:29:05.420548 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.420521    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 10:29:05.420846 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.420594    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 10:29:05.420846 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.420761    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 10:29:05.422348 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.422324    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-98krd" event={"ID":"59e913f7-bc17-4001-bed8-348466e5966c","Type":"ContainerStarted","Data":"b393ed454e3dbc7c147e8a012c303bb8344e8ad485735252d7b468d74a83bbae"}
Nov 03 10:29:05.422412 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.422383    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-98krd" event={"ID":"59e913f7-bc17-4001-bed8-348466e5966c","Type":"ContainerStarted","Data":"1997dc5450e3cb7965eb3b05b42a223f7e84a2776fa84949036a56a23de384a2"}
Nov 03 10:29:05.423588 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.423562    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kjzdq" event={"ID":"93cc76bb-77ed-493e-9207-c5bb8e30bc25","Type":"ContainerStarted","Data":"313968a59c031db374af80688503703d2d2d03ee89dfe0f05701b168f404de52"}
Nov 03 10:29:05.424641 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.424621    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zqgx" event={"ID":"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905","Type":"ContainerStarted","Data":"3e70903a7e12534ad0b15d17570f2bea210477a91ac151b60bdd2cec61dc5286"}
Nov 03 10:29:05.425298 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.425284    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 10:29:05.427090 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.427076    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8hh\" (UniqueName: \"kubernetes.io/projected/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-kube-api-access-bg8hh\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.441417 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.441378    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-crz98_51410712-fd08-4429-8d93-c0266660f335/dns-node-resolver/0.log"
Nov 03 10:29:05.518405 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.518372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.518562 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.518418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqsg\" (UniqueName: \"kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.518562 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.518539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.518644 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.518575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.518644 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.518600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.518747 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.518645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.518747 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.518720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.518858 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.518749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.519240 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.519214    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.519337 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.519228    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.519465 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.519443    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.519744 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.519728    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.519783 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.519760    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.521317 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.521298    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.521528 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.521511    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.525793 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.525771    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqsg\" (UniqueName: \"kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.539629 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.539606    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:05.568077 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:29:05.568043    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16fcb39_bfc9_4d51_af89_14d83a3d07a1.slice/crio-0b82d3e4f8156f97e9b04ebf383094b0e39d493e8ad2deacf3858e256a1c8889 WatchSource:0}: Error finding container 0b82d3e4f8156f97e9b04ebf383094b0e39d493e8ad2deacf3858e256a1c8889: Status 404 returned error can't find the container with id 0b82d3e4f8156f97e9b04ebf383094b0e39d493e8ad2deacf3858e256a1c8889
Nov 03 10:29:05.654859 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.654819    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:05.956313 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:05.956278    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-576b7767c8-btzbj"]
Nov 03 10:29:05.974648 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:29:05.969383    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a6abe6_d585_4c92_9ee8_2707962af39b.slice/crio-b285052e9cfd14cc452965839833e52aa2a82953597d8052e0e923e91c8c5579 WatchSource:0}: Error finding container b285052e9cfd14cc452965839833e52aa2a82953597d8052e0e923e91c8c5579: Status 404 returned error can't find the container with id b285052e9cfd14cc452965839833e52aa2a82953597d8052e0e923e91c8c5579
Nov 03 10:29:06.259441 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.259351    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:06.271177 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.271143    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.275622 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.275572    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 10:29:06.275804 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.275716    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 10:29:06.276870 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.276842    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 10:29:06.276980 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.276949    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fnqcd\""
Nov 03 10:29:06.277031 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.276985    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 10:29:06.277072 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.277059    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 10:29:06.277135 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.277123    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 10:29:06.277174 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.277160    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 10:29:06.277203 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.277183    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 10:29:06.277387 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.277368    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 10:29:06.278367 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.278346    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:06.428697 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.428547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.428697 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.428588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.428697 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.428617    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.428760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.428805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.428833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.428899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.428983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.429036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.429072    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.429137    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.429161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.429445 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.429261    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.430277 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.430186    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78k5n" event={"ID":"d16fcb39-bfc9-4d51-af89-14d83a3d07a1","Type":"ContainerStarted","Data":"0b82d3e4f8156f97e9b04ebf383094b0e39d493e8ad2deacf3858e256a1c8889"}
Nov 03 10:29:06.431751 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.431713    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576b7767c8-btzbj" event={"ID":"87a6abe6-d585-4c92-9ee8-2707962af39b","Type":"ContainerStarted","Data":"b285052e9cfd14cc452965839833e52aa2a82953597d8052e0e923e91c8c5579"}
Nov 03 10:29:06.444546 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.444524    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-67pqq_7532f5ee-e2ed-40a2-9d9e-277a392e1c9f/node-ca/0.log"
Nov 03 10:29:06.530489 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530616 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530485    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530616 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530616 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530616 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530791 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530791 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530646    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530791 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530791 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530791 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530962 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530962 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530843    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.530962 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.530868    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.532021 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.531681    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.532021 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.531960    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.532197 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.532022    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.533539 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.533495    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.534085 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.534037    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.534369 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.534344    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.535166 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.535033    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.535166 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.535125    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.536107 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.536083    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.536553 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.536534    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.537557 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.537521    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.537662 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.537649    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.538550 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.538530    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:06.714593 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:06.714406    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:07.031463 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.031356    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:07.170989 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.170951    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"]
Nov 03 10:29:07.179727 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.178341    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.185943 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.185583    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 10:29:07.185943 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.185664    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8520rab7os4sv\""
Nov 03 10:29:07.185943 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.185763    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 10:29:07.185943 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.185820    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"]
Nov 03 10:29:07.185943 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.185836    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 10:29:07.185943 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.185896    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 10:29:07.186315 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.186018    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gqtlm\""
Nov 03 10:29:07.188589 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.188128    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338153    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-grpc-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338241    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338347    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b87353-bd2b-4a31-a5d3-9aa7582e0354-tmp\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338378    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338425    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b87353-bd2b-4a31-a5d3-9aa7582e0354-metrics-client-ca\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.338724 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.338456    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbvbl\" (UniqueName: \"kubernetes.io/projected/54b87353-bd2b-4a31-a5d3-9aa7582e0354-kube-api-access-wbvbl\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.438713 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.438675    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerStarted","Data":"3015f5c63a4e021ad2a0c057693a9cb293a9e8524a0a8561ef113da0017915ed"}
Nov 03 10:29:07.439157 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.438937    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-grpc-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.439157 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.439009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.439157 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.439056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.439157 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.439084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.439157 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.439117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b87353-bd2b-4a31-a5d3-9aa7582e0354-tmp\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.439157 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.439149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.439461 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.439168    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.439461 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.439185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b87353-bd2b-4a31-a5d3-9aa7582e0354-metrics-client-ca\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.439461 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.439203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbvbl\" (UniqueName: \"kubernetes.io/projected/54b87353-bd2b-4a31-a5d3-9aa7582e0354-kube-api-access-wbvbl\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.440281 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.440220    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b87353-bd2b-4a31-a5d3-9aa7582e0354-metrics-client-ca\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.440803 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.440559    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b87353-bd2b-4a31-a5d3-9aa7582e0354-tmp\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.443433 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.442761    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.444679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.444346    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.444679 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.444591    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.446103 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.445725    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-grpc-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.446994 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.446748    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-98krd" event={"ID":"59e913f7-bc17-4001-bed8-348466e5966c","Type":"ContainerStarted","Data":"8eba09dbb68959b3c5352bd19af69a872b42f53603c1194654b558e53487b098"}
Nov 03 10:29:07.446994 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.446905    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.448745 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.448521    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.453640 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.453493    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbvbl\" (UniqueName: \"kubernetes.io/projected/54b87353-bd2b-4a31-a5d3-9aa7582e0354-kube-api-access-wbvbl\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.593013 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.592931    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:07.741500 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.741460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:07.741635 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.741519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:07.741635 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.741572    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:07.744730 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.744671    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:07.745377 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.745325    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:07.746169 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.746149    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:07.864593 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.864554    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:07.898730 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.898700    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:07.913170 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.913087    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"]
Nov 03 10:29:07.915844 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:29:07.915070    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b87353_bd2b_4a31_a5d3_9aa7582e0354.slice/crio-4e6497f39d5c52c7aa1e810c61867dbfb558b8077657bf04e020efb19643f4ae WatchSource:0}: Error finding container 4e6497f39d5c52c7aa1e810c61867dbfb558b8077657bf04e020efb19643f4ae: Status 404 returned error can't find the container with id 4e6497f39d5c52c7aa1e810c61867dbfb558b8077657bf04e020efb19643f4ae
Nov 03 10:29:07.923470 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:07.923433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:08.227960 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.227891    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cd6vm"]
Nov 03 10:29:08.232088 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:29:08.232043    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcaff87_f298_42b7_8fce_0850c567d9c7.slice/crio-31f108a73bb5a733253a75aef7754adf64b5c60a911f34d0734038f5be172e33 WatchSource:0}: Error finding container 31f108a73bb5a733253a75aef7754adf64b5c60a911f34d0734038f5be172e33: Status 404 returned error can't find the container with id 31f108a73bb5a733253a75aef7754adf64b5c60a911f34d0734038f5be172e33
Nov 03 10:29:08.253151 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.253122    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wm4x2"]
Nov 03 10:29:08.286696 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.286636    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w4txd"]
Nov 03 10:29:08.453615 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.453530    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zqgx" event={"ID":"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905","Type":"ContainerStarted","Data":"3ddd76d6ccc4d95fc82d21e011937f242ce47c353d1ed27e67b2e6c05bfca350"}
Nov 03 10:29:08.453615 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.453597    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9zqgx" event={"ID":"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905","Type":"ContainerStarted","Data":"8ce1aa39e94672fc77ebf832bd370c467ea73fe9ffaeb3e3bda639ea0bd72efb"}
Nov 03 10:29:08.454474 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.454454    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:08.456959 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.456893    2573 generic.go:356] "Generic (PLEG): container finished" podID="d16fcb39-bfc9-4d51-af89-14d83a3d07a1" containerID="b7509b55c8c2b1e21c0d5e92cab3fcfd60a6badce3ab9f8be10fcd867cf4a9fb" exitCode=0
Nov 03 10:29:08.456959 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.456947    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78k5n" event={"ID":"d16fcb39-bfc9-4d51-af89-14d83a3d07a1","Type":"ContainerDied","Data":"b7509b55c8c2b1e21c0d5e92cab3fcfd60a6badce3ab9f8be10fcd867cf4a9fb"}
Nov 03 10:29:08.458715 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.458692    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wm4x2" event={"ID":"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b","Type":"ContainerStarted","Data":"136f55de77188ed0db84e1b9d019b11cd7cbacadff8cca77d8c3550722ff2335"}
Nov 03 10:29:08.460133 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.460110    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w4txd" event={"ID":"430e1cec-2732-4742-8eed-d2ba0fc4caf3","Type":"ContainerStarted","Data":"a43122507a4f5325f86d1f6ed224d4e78ca3e1c0233504474f90a536d14d60af"}
Nov 03 10:29:08.461621 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.461597    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cd6vm" event={"ID":"cbcaff87-f298-42b7-8fce-0850c567d9c7","Type":"ContainerStarted","Data":"31f108a73bb5a733253a75aef7754adf64b5c60a911f34d0734038f5be172e33"}
Nov 03 10:29:08.463412 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.463389    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf" event={"ID":"54b87353-bd2b-4a31-a5d3-9aa7582e0354","Type":"ContainerStarted","Data":"4e6497f39d5c52c7aa1e810c61867dbfb558b8077657bf04e020efb19643f4ae"}
Nov 03 10:29:08.477496 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:08.477431    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9zqgx" podStartSLOduration=1.677789158 podStartE2EDuration="4.477412932s" totalImagesPullingTime="2.799623774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:08.475767175 +0000 UTC m=+33.884659159" watchObservedRunningTime="2026-11-03 10:29:08.477412932 +0000 UTC m=+33.886304917"
Nov 03 10:29:09.480807 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.480756    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78k5n" event={"ID":"d16fcb39-bfc9-4d51-af89-14d83a3d07a1","Type":"ContainerStarted","Data":"505f468989635803f686d2265781999340501277ee28d68dcfb06248f00f2876"}
Nov 03 10:29:09.481167 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.480855    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78k5n" event={"ID":"d16fcb39-bfc9-4d51-af89-14d83a3d07a1","Type":"ContainerStarted","Data":"1a54266202ed06ce3d12db970535124c0a2f28d97daa5814a5b124aa908599c3"}
Nov 03 10:29:09.502105 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.502035    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-78k5n" podStartSLOduration=2.519732726 podStartE2EDuration="4.502018068s" totalImagesPullingTime="1.852309502s" totalInitContainerRuntime="129.97584ms" isStatefulPod=true podCreationTimestamp="2026-11-03 10:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:09.500808539 +0000 UTC m=+34.909700523" watchObservedRunningTime="2026-11-03 10:29:09.502018068 +0000 UTC m=+34.910910054"
Nov 03 10:29:09.593444 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.593411    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-767d4fbf8c-86qwm"]
Nov 03 10:29:09.613227 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.613143    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-767d4fbf8c-86qwm"]
Nov 03 10:29:09.613369 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.613253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.617568 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.617547    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 10:29:09.617718 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.617679    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 10:29:09.643542 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.643511    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 10:29:09.643884 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.643765    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-30l0kuump9kjj\""
Nov 03 10:29:09.643884 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.643850    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7whcj\""
Nov 03 10:29:09.644058 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.643900    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 10:29:09.759859 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.759818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/11b7246b-031f-46cd-ab96-814d2e5d4a72-audit-log\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.760042 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.759870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-client-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.760042 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.759898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-tls\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.760042 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.759945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-client-certs\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.760042 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.759961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4wd\" (UniqueName: \"kubernetes.io/projected/11b7246b-031f-46cd-ab96-814d2e5d4a72-kube-api-access-fs4wd\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.760042 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.759981    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-metrics-server-audit-profiles\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.760042 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.760007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.861491 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.861448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/11b7246b-031f-46cd-ab96-814d2e5d4a72-audit-log\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.861665 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.861522    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-client-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.861665 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.861559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-tls\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.861665 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.861602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-client-certs\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.861665 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.861628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4wd\" (UniqueName: \"kubernetes.io/projected/11b7246b-031f-46cd-ab96-814d2e5d4a72-kube-api-access-fs4wd\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.861665 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.861655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-metrics-server-audit-profiles\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.861951 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.861714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.862713 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.862686    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.863015 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.862981    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/11b7246b-031f-46cd-ab96-814d2e5d4a72-audit-log\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.864735 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.864688    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-metrics-server-audit-profiles\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.867317 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.867257    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-client-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.868261 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.868181    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-tls\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.868721 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.868703    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-client-certs\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.883540 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.883516    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4wd\" (UniqueName: \"kubernetes.io/projected/11b7246b-031f-46cd-ab96-814d2e5d4a72-kube-api-access-fs4wd\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.950032 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.949999    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:09.972440 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.971340    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"]
Nov 03 10:29:09.989479 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.989448    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"]
Nov 03 10:29:09.989877 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.989556    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:09.993623 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.993601    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 10:29:09.993940 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:09.993887    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c9gx6\""
Nov 03 10:29:10.164598 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.164514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4q2\" (UniqueName: \"kubernetes.io/projected/f9634e11-0c6c-4832-95a9-6665d679e1fd-kube-api-access-fn4q2\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:10.164775 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.164615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9634e11-0c6c-4832-95a9-6665d679e1fd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:10.265455 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.265419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4q2\" (UniqueName: \"kubernetes.io/projected/f9634e11-0c6c-4832-95a9-6665d679e1fd-kube-api-access-fn4q2\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:10.265624 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.265476    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9634e11-0c6c-4832-95a9-6665d679e1fd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:10.275773 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.275744    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4q2\" (UniqueName: \"kubernetes.io/projected/f9634e11-0c6c-4832-95a9-6665d679e1fd-kube-api-access-fn4q2\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:10.275994 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.275971    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9634e11-0c6c-4832-95a9-6665d679e1fd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:10.331038 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.330999    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:10.494215 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.493058    2573 generic.go:356] "Generic (PLEG): container finished" podID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerID="1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c" exitCode=0
Nov 03 10:29:10.494215 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.493867    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerDied","Data":"1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c"}
Nov 03 10:29:10.515080 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.512783    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-98krd" event={"ID":"59e913f7-bc17-4001-bed8-348466e5966c","Type":"ContainerStarted","Data":"db6bc484582ad4f360f59be6e94e2abba4f4b2e67084f77286b8a4085760b92d"}
Nov 03 10:29:10.537948 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.536192    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-98krd" podStartSLOduration=2.398035525 podStartE2EDuration="6.536171741s" totalImagesPullingTime="4.138136216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:29:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:10.53446408 +0000 UTC m=+35.943356067" watchObservedRunningTime="2026-11-03 10:29:10.536171741 +0000 UTC m=+35.945063726"
Nov 03 10:29:10.735882 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.735761    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"]
Nov 03 10:29:10.770343 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:10.769185    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-767d4fbf8c-86qwm"]
Nov 03 10:29:11.524085 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.524046    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kjzdq" event={"ID":"93cc76bb-77ed-493e-9207-c5bb8e30bc25","Type":"ContainerStarted","Data":"6890c5f4162fa29179746ef26109125f7c7130520ae63c501c8e53598a507083"}
Nov 03 10:29:11.532473 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.532405    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6" event={"ID":"f9634e11-0c6c-4832-95a9-6665d679e1fd","Type":"ContainerStarted","Data":"829066a0ec2fa0b403b17eb3991f126d04587cbb459efb11898864b3b69059e6"}
Nov 03 10:29:11.533863 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.533797    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm" event={"ID":"11b7246b-031f-46cd-ab96-814d2e5d4a72","Type":"ContainerStarted","Data":"6f5988790763ab7c1bc0394604de6b74d91b0c6c1ccc605043910d53b790bc80"}
Nov 03 10:29:11.536516 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.535954    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576b7767c8-btzbj" event={"ID":"87a6abe6-d585-4c92-9ee8-2707962af39b","Type":"ContainerStarted","Data":"0f4d85593b948e960513f2853ced307f1669c7e68bf48ad652b9302839470fa9"}
Nov 03 10:29:11.540435 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.540301    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kjzdq" podStartSLOduration=2.125365624 podStartE2EDuration="7.54028644s" totalImagesPullingTime="5.414920816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:11.538444077 +0000 UTC m=+36.947336062" watchObservedRunningTime="2026-11-03 10:29:11.54028644 +0000 UTC m=+36.949178425"
Nov 03 10:29:11.543574 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.543499    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wm4x2" event={"ID":"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b","Type":"ContainerStarted","Data":"5704f4342b91ed1457983b8881bf4402a8adbef08a85b2c5cedf3e13bfb35061"}
Nov 03 10:29:11.543574 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.543541    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wm4x2" event={"ID":"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b","Type":"ContainerStarted","Data":"17ded215081e5fabf0f7a0b9608a2323faefd02615e5b91bac6dfe785154ddb8"}
Nov 03 10:29:11.556169 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.555764    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-576b7767c8-btzbj" podStartSLOduration=1.164803664 podStartE2EDuration="6.555746978s" totalImagesPullingTime="5.390943314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:11.554127312 +0000 UTC m=+36.963019297" watchObservedRunningTime="2026-11-03 10:29:11.555746978 +0000 UTC m=+36.964638964"
Nov 03 10:29:11.573058 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:11.572181    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wm4x2" podStartSLOduration=34.385290618 podStartE2EDuration="36.572162425s" totalImagesPullingTime="2.186871807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:11.571373599 +0000 UTC m=+36.980265584" watchObservedRunningTime="2026-11-03 10:29:11.572162425 +0000 UTC m=+36.981054411"
Nov 03 10:29:12.490037 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.488909    2573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-576b7767c8-btzbj"]
Nov 03 10:29:12.519033 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.518982    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59787cccbc-9wk85"]
Nov 03 10:29:12.537840 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.537809    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59787cccbc-9wk85"]
Nov 03 10:29:12.538307 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.537938    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.551398 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.551372    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf" event={"ID":"54b87353-bd2b-4a31-a5d3-9aa7582e0354","Type":"ContainerStarted","Data":"38e9cd641faa70744d7734022cb5f5089786e5c9e747e52fd0d1deb7e7c04d17"}
Nov 03 10:29:12.551508 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.551440    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf" event={"ID":"54b87353-bd2b-4a31-a5d3-9aa7582e0354","Type":"ContainerStarted","Data":"abdd9c03e147e5d43c4cef279cf2a97cba9bd9560a02c52574e877c22404c3a6"}
Nov 03 10:29:12.698156 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.698119    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.698342 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.698172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.698508 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.698481    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.698602 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.698531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.698825 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.698776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.699061 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.699042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.699122 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.699077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m69mv\" (UniqueName: \"kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.699165 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.699126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.799584 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.799504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.799584 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.799539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69mv\" (UniqueName: \"kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.799584 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.799558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.799584 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.799575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.799900 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.799599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.799900 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.799619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.799900 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.799638    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.799900 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.799667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.800982 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.800955    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.801261 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.801227    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.801261 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.801238    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.802028 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.801808    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.802200 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.802157    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.805273 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.804348    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.806209 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.806187    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.809878 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.809844    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69mv\" (UniqueName: \"kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:12.883646 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:12.883358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:13.198864 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:13.198827    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59787cccbc-9wk85"]
Nov 03 10:29:13.253440 ip-10-0-134-12 kubenswrapper[2573]: W1103 10:29:13.253403    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856fad64_8c12_4623_b444_5d99425a7505.slice/crio-f2c19c519c02b28a65f483c6dccbbe9b22e3d77b809e98972034f7198ea71974 WatchSource:0}: Error finding container f2c19c519c02b28a65f483c6dccbbe9b22e3d77b809e98972034f7198ea71974: Status 404 returned error can't find the container with id f2c19c519c02b28a65f483c6dccbbe9b22e3d77b809e98972034f7198ea71974
Nov 03 10:29:13.562537 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:13.562450    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf" event={"ID":"54b87353-bd2b-4a31-a5d3-9aa7582e0354","Type":"ContainerStarted","Data":"2b986bef07eedc3b3509b783733bb83d0f3d5e376f61c79ae2a6e708b86fb88d"}
Nov 03 10:29:13.564578 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:13.564542    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59787cccbc-9wk85" event={"ID":"856fad64-8c12-4623-b444-5d99425a7505","Type":"ContainerStarted","Data":"7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750"}
Nov 03 10:29:13.564709 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:13.564594    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59787cccbc-9wk85" event={"ID":"856fad64-8c12-4623-b444-5d99425a7505","Type":"ContainerStarted","Data":"f2c19c519c02b28a65f483c6dccbbe9b22e3d77b809e98972034f7198ea71974"}
Nov 03 10:29:13.580957 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:13.580876    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59787cccbc-9wk85" podStartSLOduration=1.5808624500000001 podStartE2EDuration="1.58086245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:13.580535242 +0000 UTC m=+38.989427226" watchObservedRunningTime="2026-11-03 10:29:13.58086245 +0000 UTC m=+38.989754434"
Nov 03 10:29:14.570813 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:14.570784    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm" event={"ID":"11b7246b-031f-46cd-ab96-814d2e5d4a72","Type":"ContainerStarted","Data":"659feffa76df66bda0041fb5af3f83d1cd1e552fc98b7228d2f6ac41f106db10"}
Nov 03 10:29:14.591892 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:14.591850    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm" podStartSLOduration=2.689663079 podStartE2EDuration="5.591836722s" totalImagesPullingTime="2.902173643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:14.59021572 +0000 UTC m=+39.999107705" watchObservedRunningTime="2026-11-03 10:29:14.591836722 +0000 UTC m=+40.000728705"
Nov 03 10:29:15.575606 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.575571    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cd6vm" event={"ID":"cbcaff87-f298-42b7-8fce-0850c567d9c7","Type":"ContainerStarted","Data":"633d8b26100453e660d3a31e25af450c8118a047de84c03d285d4210a4a77145"}
Nov 03 10:29:15.576094 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.575770    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:15.578645 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.578621    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf" event={"ID":"54b87353-bd2b-4a31-a5d3-9aa7582e0354","Type":"ContainerStarted","Data":"ea33ab5a9358bbe564faf557522dc8fbda4ce51db3da7b507ec14f35c996c587"}
Nov 03 10:29:15.578769 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.578683    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf" event={"ID":"54b87353-bd2b-4a31-a5d3-9aa7582e0354","Type":"ContainerStarted","Data":"d68712aec60235361124ebb889875c171e91501ba61546bdcda2fe1667ab28ff"}
Nov 03 10:29:15.578769 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.578722    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:15.578769 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.578749    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf" event={"ID":"54b87353-bd2b-4a31-a5d3-9aa7582e0354","Type":"ContainerStarted","Data":"fe93f334d413d24d1b9e4785fdca2fa30577f3178806c1aca6efd3530b07b897"}
Nov 03 10:29:15.580264 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.580242    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6" event={"ID":"f9634e11-0c6c-4832-95a9-6665d679e1fd","Type":"ContainerStarted","Data":"d44b79104fb8321fed75411c15d27145199c6a0fb9cd2866872584718871d2c9"}
Nov 03 10:29:15.580383 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.580361    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:15.583992 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.583972    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerStarted","Data":"5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a"}
Nov 03 10:29:15.584090 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.584046    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerStarted","Data":"50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd"}
Nov 03 10:29:15.584161 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.584131    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerStarted","Data":"0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45"}
Nov 03 10:29:15.584214 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.584174    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerStarted","Data":"9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee"}
Nov 03 10:29:15.584263 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.584220    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerStarted","Data":"7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754"}
Nov 03 10:29:15.584311 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.584264    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerStarted","Data":"593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004"}
Nov 03 10:29:15.585721 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.585701    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:15.594011 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.593975    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cd6vm" podStartSLOduration=34.256307918 podStartE2EDuration="40.593962491s" totalImagesPullingTime="6.337654573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:15.593068829 +0000 UTC m=+41.001960804" watchObservedRunningTime="2026-11-03 10:29:15.593962491 +0000 UTC m=+41.002854474"
Nov 03 10:29:15.615209 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.615164    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf" podStartSLOduration=2.359407556 podStartE2EDuration="8.615146613s" totalImagesPullingTime="6.255739057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:15.613984872 +0000 UTC m=+41.022876859" watchObservedRunningTime="2026-11-03 10:29:15.615146613 +0000 UTC m=+41.024038594"
Nov 03 10:29:15.640560 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.640518    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.255227292 podStartE2EDuration="9.64050119s" totalImagesPullingTime="6.35844689s" totalInitContainerRuntime="26.827008ms" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:15.638610471 +0000 UTC m=+41.047502455" watchObservedRunningTime="2026-11-03 10:29:15.64050119 +0000 UTC m=+41.049393175"
Nov 03 10:29:15.655473 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.655442    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:15.658692 ip-10-0-134-12 kubenswrapper[2573]: I1103 10:29:15.657578    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6" podStartSLOduration=2.847413534 podStartE2EDuration="6.657561002s" totalImagesPullingTime="3.810147468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:15.657171778 +0000 UTC m=+41.066063763" watchObservedRunningTime="2026-11-03 10:29:15.657561002 +0000 UTC m=+41.066452987"
Nov 03 10:29:16.035629 ip-10-0-134-12 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 10:29:16.050358 ip-10-0-134-12 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 10:29:16.050567 ip-10-0-134-12 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 10:29:16.050625 ip-10-0-134-12 systemd[1]: kubelet.service: Consumed 3.191s CPU time.
Nov 03 10:29:16.061526 ip-10-0-134-12 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 10:29:16.144235 ip-10-0-134-12 kubenswrapper[5393]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:29:16.144235 ip-10-0-134-12 kubenswrapper[5393]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 10:29:16.144235 ip-10-0-134-12 kubenswrapper[5393]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:29:16.144235 ip-10-0-134-12 kubenswrapper[5393]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147578    5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147669    5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147673    5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147676    5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147679    5393 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147682    5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147685    5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147688    5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147690    5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:16.147682 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147693    5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147696    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147699    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147701    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147704    5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147706    5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147711    5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147715    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147718    5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147721    5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147724    5393 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147726    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147729    5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147732    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147734    5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147737    5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147740    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147743    5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147746    5393 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:16.148032 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147749    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147751    5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147754    5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147756    5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147759    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147762    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147765    5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147767    5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147770    5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147772    5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147775    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147777    5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147780    5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147782    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147785    5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147787    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147790    5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147792    5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147795    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:16.148520 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147797    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147801    5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147804    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147806    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147809    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147811    5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147814    5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147816    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147819    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147823    5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147827    5393 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147830    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147834    5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147836    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147839    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147841    5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147844    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147847    5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147849    5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:16.149003 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147852    5393 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147854    5393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147857    5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147859    5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147862    5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147865    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147867    5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147870    5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147872    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147875    5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147877    5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147880    5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147882    5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147889    5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147891    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147894    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147896    5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147900    5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147903    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:16.149456 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147906    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147908    5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147911    5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147928    5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147931    5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147934    5393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147937    5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147940    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147942    5393 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147945    5393 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147948    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147950    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147953    5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147956    5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147959    5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147961    5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147964    5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147967    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147969    5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147971    5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.147974    5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:16.150018 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148736    5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148743    5393 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148746    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148749    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148751    5393 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148757    5393 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148760    5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148762    5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148765    5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148768    5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148771    5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148774    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148776    5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148779    5393 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148784    5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148788    5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148792    5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148795    5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148797    5393 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148800    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:16.150523 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148803    5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148805    5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148808    5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148810    5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148813    5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148815    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148818    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148821    5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148823    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148826    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148829    5393 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148831    5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148834    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148836    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148839    5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148841    5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148844    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148847    5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148850    5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148852    5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:16.151083 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148855    5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148858    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148860    5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148863    5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148866    5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148869    5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148871    5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148874    5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148877    5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148880    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148882    5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148885    5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148887    5393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148890    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148892    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148895    5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148898    5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148900    5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148903    5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:16.151559 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148905    5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148908    5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148910    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148913    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148930    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148933    5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148935    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148938    5393 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148941    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148943    5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148947    5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148949    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148952    5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148954    5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148957    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148960    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148963    5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148966    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148969    5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:16.152075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148972    5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148974    5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148977    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148980    5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148982    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148985    5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148987    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148990    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148993    5393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148995    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.148998    5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149001    5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149003    5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149006    5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149009    5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149011    5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149014    5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149017    5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149020    5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149024    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:16.152566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149027    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149029    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149032    5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149036    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149039    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149042    5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149045    5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149048    5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149537    5393 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149542    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149546    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149549    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149551    5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149554    5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149557    5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149560    5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149562    5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149565    5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149567    5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:16.153066 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149570    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149572    5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149575    5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149578    5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149580    5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149583    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149585    5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149588    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149590    5393 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149593    5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149596    5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149598    5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149601    5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149604    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149607    5393 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149609    5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149613    5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149617    5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149621    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:16.153531 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149624    5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149628    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149631    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149634    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149637    5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149640    5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149643    5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149645    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149647    5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149650    5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149652    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149655    5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149657    5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149660    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149662    5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149664    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149667    5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149669    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149672    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:16.154025 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149674    5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149677    5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149688    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149691    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149694    5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149697    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149700    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149702    5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149705    5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149707    5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149711    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149714    5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149716    5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149721    5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149725    5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149728    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149730    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149734    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149736    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:16.154461 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149740    5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149743    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149745    5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149748    5393 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149751    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149753    5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149755    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149758    5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149760    5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149763    5393 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149765    5393 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149767    5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149770    5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149772    5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149775    5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149778    5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149780    5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149783    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149785    5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149788    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:16.154906 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149790    5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149792    5393 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149795    5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149799    5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149801    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149804    5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149806    5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149808    5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149811    5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149814    5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149816    5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149819    5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149821    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149824    5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149827    5393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149829    5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149832    5393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.149836    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149947    5393 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149961    5393 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149968    5393 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 10:29:16.155399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149973    5393 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149978    5393 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149981    5393 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149990    5393 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149995    5393 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.149998    5393 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150001    5393 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150004    5393 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150007    5393 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150010    5393 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150014    5393 flags.go:64] FLAG: --cgroup-root=""
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150016    5393 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150019    5393 flags.go:64] FLAG: --client-ca-file=""
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150022    5393 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150027    5393 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150032    5393 flags.go:64] FLAG: --cluster-domain=""
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150035    5393 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150038    5393 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150041    5393 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150044    5393 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150048    5393 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150051    5393 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150054    5393 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 10:29:16.155896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150057    5393 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150060    5393 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150064    5393 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150066    5393 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150069    5393 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150072    5393 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150077    5393 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150080    5393 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150083    5393 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150085    5393 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150088    5393 flags.go:64] FLAG: --enable-server="true"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150091    5393 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150096    5393 flags.go:64] FLAG: --event-burst="100"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150099    5393 flags.go:64] FLAG: --event-qps="50"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150102    5393 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150105    5393 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150108    5393 flags.go:64] FLAG: --eviction-hard=""
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150112    5393 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150116    5393 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150119    5393 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150122    5393 flags.go:64] FLAG: --eviction-soft=""
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150124    5393 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150127    5393 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150132    5393 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150135    5393 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 10:29:16.156466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150138    5393 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150141    5393 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150144    5393 flags.go:64] FLAG: --feature-gates=""
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150152    5393 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150155    5393 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150158    5393 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150161    5393 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150164    5393 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150167    5393 flags.go:64] FLAG: --help="false"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150170    5393 flags.go:64] FLAG: --hostname-override="ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150176    5393 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150179    5393 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150182    5393 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150186    5393 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150189    5393 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150192    5393 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150195    5393 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150198    5393 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150200    5393 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150203    5393 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150206    5393 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150209    5393 flags.go:64] FLAG: --kube-reserved=""
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150212    5393 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150215    5393 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 10:29:16.157118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150218    5393 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150220    5393 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150223    5393 flags.go:64] FLAG: --lock-file=""
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150226    5393 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150229    5393 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150232    5393 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150240    5393 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150242    5393 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150246    5393 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150249    5393 flags.go:64] FLAG: --logging-format="text"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150251    5393 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150255    5393 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150257    5393 flags.go:64] FLAG: --manifest-url=""
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150260    5393 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150264    5393 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150267    5393 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150271    5393 flags.go:64] FLAG: --max-pods="110"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150274    5393 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150279    5393 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150283    5393 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150286    5393 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150289    5393 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150292    5393 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150295    5393 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150301    5393 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 10:29:16.157717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150304    5393 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150307    5393 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150310    5393 flags.go:64] FLAG: --pod-cidr=""
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150313    5393 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150316    5393 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150319    5393 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150322    5393 flags.go:64] FLAG: --port="10250"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150325    5393 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150328    5393 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d381cf900aeee9ec"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150331    5393 flags.go:64] FLAG: --qos-reserved=""
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150334    5393 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150337    5393 flags.go:64] FLAG: --register-node="true"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150340    5393 flags.go:64] FLAG: --register-with-taints=""
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150350    5393 flags.go:64] FLAG: --registry-burst="10"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150353    5393 flags.go:64] FLAG: --registry-qps="5"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150356    5393 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150359    5393 flags.go:64] FLAG: --reserved-memory=""
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150363    5393 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150366    5393 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150369    5393 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150371    5393 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150374    5393 flags.go:64] FLAG: --runonce="false"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150377    5393 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150380    5393 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150383    5393 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150387    5393 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150390    5393 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 10:29:16.158390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150393    5393 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150396    5393 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150399    5393 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150403    5393 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150406    5393 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150409    5393 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150412    5393 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150415    5393 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150418    5393 flags.go:64] FLAG: --system-cgroups=""
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150421    5393 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150426    5393 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150429    5393 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150432    5393 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150436    5393 flags.go:64] FLAG: --tls-min-version=""
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150439    5393 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150442    5393 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150444    5393 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150448    5393 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150452    5393 flags.go:64] FLAG: --v="2"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150456    5393 flags.go:64] FLAG: --version="false"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150460    5393 flags.go:64] FLAG: --vmodule=""
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150464    5393 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150467    5393 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150595    5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:16.159099 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150598    5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150601    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150604    5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150606    5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150609    5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150611    5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150615    5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150617    5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150620    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150623    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150625    5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150628    5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150630    5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150633    5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150654    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150658    5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150662    5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150665    5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150668    5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150671    5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:16.160118 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150674    5393 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150676    5393 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150679    5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150682    5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150684    5393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150687    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150690    5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150692    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150695    5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150698    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150700    5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150703    5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150705    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150708    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150710    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150713    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150716    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150719    5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150721    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:16.161219 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150724    5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150726    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150729    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150732    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150734    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150737    5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150740    5393 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150742    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150744    5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150747    5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150749    5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150752    5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150755    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150757    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150759    5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150762    5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150765    5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150767    5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150769    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:16.162210 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150772    5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150776    5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150780    5393 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150783    5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150786    5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150790    5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150793    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150796    5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150799    5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150801    5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150804    5393 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150806    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150809    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150812    5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150814    5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150817    5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150819    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150827    5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150830    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:16.162802 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150833    5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150835    5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150837    5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150840    5393 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150842    5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150845    5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150847    5393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150849    5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150852    5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150855    5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150857    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150860    5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150862    5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150865    5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150867    5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150870    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150872    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150874    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150877    5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150880    5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:16.163352 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150882    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150884    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150887    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150889    5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150891    5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150894    5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150897    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150899    5393 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.150939    5393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150977    5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150982    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150988    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150991    5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150994    5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150996    5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.150999    5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:16.163936 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151001    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151004    5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151008    5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151012    5393 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151016    5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151019    5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151022    5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151026    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151029    5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151031    5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151034    5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151037    5393 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151039    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151042    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151047    5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151049    5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151052    5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151054    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151057    5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:16.164507 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151059    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151062    5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151064    5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151067    5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151069    5393 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151072    5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151074    5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151076    5393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151079    5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151082    5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151084    5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151087    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151090    5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151093    5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151095    5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151098    5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151101    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151107    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151110    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151112    5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:16.165217 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151115    5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151117    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151120    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151122    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151125    5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151127    5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151130    5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151133    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151136    5393 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151138    5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151141    5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151143    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151146    5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151149    5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151151    5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151153    5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151156    5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151159    5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151161    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151164    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:16.165707 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151166    5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151169    5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151179    5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151182    5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151185    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151187    5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151190    5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151192    5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151194    5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151198    5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151200    5393 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151203    5393 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151205    5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151209    5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151211    5393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151213    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151216    5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151218    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151221    5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:16.166201 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151226    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151229    5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151232    5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151234    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151237    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151240    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151242    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151245    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151247    5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151249    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151252    5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151254    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151257    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151259    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151262    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151264    5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151273    5393 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151276    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151279    5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:16.166638 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151281    5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.151284    5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.155150    5393 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.155164    5393 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.155182    5393 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.155187    5393 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155244    5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155249    5393 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155252    5393 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155256    5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155259    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155261    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155264    5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155267    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155270    5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155272    5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155275    5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155277    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155280    5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155282    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:16.167180 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155285    5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155288    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155291    5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155294    5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155297    5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155299    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155302    5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155304    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155307    5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155310    5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155313    5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155316    5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155318    5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155322    5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155326    5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155329    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155332    5393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155335    5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155337    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:16.167789 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155339    5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155342    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155344    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155347    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155350    5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155352    5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155355    5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155357    5393 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155360    5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155363    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155365    5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155368    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155370    5393 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155373    5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155376    5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155379    5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155381    5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155383    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155386    5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155389    5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:16.168273 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155391    5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155393    5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155396    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155400    5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155404    5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155407    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155410    5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155412    5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155415    5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155417    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155420    5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155422    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155425    5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155427    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155430    5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155432    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155435    5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155438    5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155440    5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:16.168743 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155442    5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155445    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155447    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155449    5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155452    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155454    5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155457    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155460    5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155462    5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155465    5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155467    5393 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155469    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155472    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155474    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155477    5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155479    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155482    5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155484    5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155487    5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155489    5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:16.169245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155492    5393 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155494    5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155497    5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155499    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155502    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155504    5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155506    5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155509    5393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155511    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155514    5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155516    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155519    5393 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155521    5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155524    5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.155549    5393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155684    5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:16.169734 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155689    5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155691    5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155694    5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155697    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155699    5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155702    5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155705    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155707    5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155709    5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155712    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155714    5393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155717    5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155719    5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155722    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155725    5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155727    5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155730    5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155732    5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155735    5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:16.170145 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155737    5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155740    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155742    5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155745    5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155747    5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155750    5393 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155752    5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155755    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155757    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155760    5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155762    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155764    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155767    5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155769    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155772    5393 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155774    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155777    5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155779    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155782    5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155784    5393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:16.170598 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155787    5393 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155789    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155791    5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155794    5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155796    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155799    5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155801    5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155803    5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155807    5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155809    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155812    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155814    5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155816    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155819    5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155821    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155824    5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155826    5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155828    5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155831    5393 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:16.171187 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155833    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155836    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155838    5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155840    5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155843    5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155845    5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155848    5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155851    5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155854    5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155858    5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155861    5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155864    5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155866    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155868    5393 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155871    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155873    5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155876    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155878    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155880    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:16.171664 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155883    5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155885    5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155887    5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155890    5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155893    5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155895    5393 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155897    5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155901    5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155905    5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155909    5393 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155912    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155931    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155935    5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155941    5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155943    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155946    5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155950    5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155953    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155956    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155958    5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:16.172173 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155961    5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155964    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155966    5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155969    5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155971    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155973    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155976    5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.155978    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.156002    5393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156036    5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156041    5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156044    5393 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156047    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156050    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156053    5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156055    5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 10:29:16.173000 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156058    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156061    5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156063    5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156066    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156068    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156071    5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156073    5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156076    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156079    5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156081    5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156087    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156089    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156096    5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156098    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156101    5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156103    5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156106    5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156109    5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156111    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 10:29:16.173405 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156114    5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156116    5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156119    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156121    5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156124    5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156126    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156129    5393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156131    5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156133    5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156136    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156138    5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156141    5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156143    5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156146    5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156149    5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156152    5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156154    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156157    5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156159    5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 10:29:16.173861 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156162    5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156164    5393 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156167    5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156169    5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156172    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156175    5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156178    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156181    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156183    5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156186    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156188    5393 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156191    5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156193    5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156196    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156198    5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156201    5393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156203    5393 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156206    5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156208    5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156210    5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 10:29:16.174392 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156214    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156216    5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156219    5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156221    5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156224    5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156226    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156229    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156231    5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156234    5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156237    5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156239    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156241    5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156244    5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156246    5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156249    5393 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156251    5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156255    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156257    5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156261    5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 10:29:16.174865 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156263    5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156266    5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156268    5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156271    5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156276    5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156279    5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156282    5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156285    5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156287    5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156290    5393 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156292    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156294    5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156297    5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156299    5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156302    5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156304    5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156307    5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156309    5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156311    5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156314    5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 10:29:16.175335 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156316    5393 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:16.156319    5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.156366    5393 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.157314    5393 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.157364    5393 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.157537    5393 server.go:1014] "Starting client certificate rotation"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.157676    5393 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.157733    5393 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 10:23:34 +0000 UTC" deadline="2028-08-11 01:29:15.845883946 +0000 UTC"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.157760    5393 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15518h59m59.68812576s"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.158030    5393 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.159178    5393 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.161876    5393 log.go:25] "Validated CRI v1 runtime API"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.163797    5393 log.go:25] "Validated CRI v1 image API"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.164592    5393 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 10:29:16.175801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.166721    5393 fs.go:135] Filesystem UUIDs: map[546f2653-f94d-4dc3-9c48-08cebc0ad93a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d44f2494-a1b2-4252-8156-f0b821cbbd13:/dev/nvme0n1p3]
Nov 03 10:29:16.176271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.166743    5393 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b82d3e4f8156f97e9b04ebf383094b0e39d493e8ad2deacf3858e256a1c8889/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b82d3e4f8156f97e9b04ebf383094b0e39d493e8ad2deacf3858e256a1c8889/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/136a60959a855bfbeff212a5c5e618b6968c28f5b662533fc576518fe14e4e38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/136a60959a855bfbeff212a5c5e618b6968c28f5b662533fc576518fe14e4e38/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/136f55de77188ed0db84e1b9d019b11cd7cbacadff8cca77d8c3550722ff2335/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/136f55de77188ed0db84e1b9d019b11cd7cbacadff8cca77d8c3550722ff2335/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/161058e45d6663ee796adea96ae77c39d8bec7a3153b59883d19651862dc197c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/161058e45d6663ee796adea96ae77c39d8bec7a3153b59883d19651862dc197c/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1997dc5450e3cb7965eb3b05b42a223f7e84a2776fa84949036a56a23de384a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1997dc5450e3cb7965eb3b05b42a223f7e84a2776fa84949036a56a23de384a2/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3015f5c63a4e021ad2a0c057693a9cb293a9e8524a0a8561ef113da0017915ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3015f5c63a4e021ad2a0c057693a9cb293a9e8524a0a8561ef113da0017915ed/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/313968a59c031db374af80688503703d2d2d03ee89dfe0f05701b168f404de52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/313968a59c031db374af80688503703d2d2d03ee89dfe0f05701b168f404de52/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31eb62fa6614e04ced7dc4a8147116968a2b7343fa2d787f6c815248b9136fa1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31eb62fa6614e04ced7dc4a8147116968a2b7343fa2d787f6c815248b9136fa1/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31f108a73bb5a733253a75aef7754adf64b5c60a911f34d0734038f5be172e33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31f108a73bb5a733253a75aef7754adf64b5c60a911f34d0734038f5be172e33/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e70903a7e12534ad0b15d17570f2bea210477a91ac151b60bdd2cec61dc5286/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e70903a7e12534ad0b15d17570f2bea210477a91ac151b60bdd2cec61dc5286/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e6497f39d5c52c7aa1e810c61867dbfb558b8077657bf04e020efb19643f4ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e6497f39d5c52c7aa1e810c61867dbfb558b8077657bf04e020efb19643f4ae/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53f0ab87129c8ba04c72fa95466ee09074e903d5635d7eaf2aa0840957ce8578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53f0ab87129c8ba04c72fa95466ee09074e903d5635d7eaf2aa0840957ce8578/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f5988790763ab7c1bc0394604de6b74d91b0c6c1ccc605043910d53b790bc80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f5988790763ab7c1bc0394604de6b74d91b0c6c1ccc605043910d53b790bc80/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8115cbe0c0262d974bb9566912e76cc706913703b7a4bd2194018a99a7fb0b64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8115cbe0c0262d974bb9566912e76cc706913703b7a4bd2194018a99a7fb0b64/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/829066a0ec2fa0b403b17eb3991f126d04587cbb459efb11898864b3b69059e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/829066a0ec2fa0b403b17eb3991f126d04587cbb459efb11898864b3b69059e6/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91a30875268765bed1cc83c1bdc52cf8b049b9032f79a710d0ad0eeceeb86c5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91a30875268765bed1cc83c1bdc52cf8b049b9032f79a710d0ad0eeceeb86c5e/userdata/shm major:0 minor:57 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2f8a0a98ec816ccc44f3d6ed26a5bd85ecc09db14a1226215d142e04c236257/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2f8a0a98ec816ccc44f3d6ed26a5bd85ecc09db14a1226215d142e04c236257/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a43122507a4f5325f86d1f6ed224d4e78ca3e1c0233504474f90a536d14d60af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a43122507a4f5325f86d1f6ed224d4e78ca3e1c0233504474f90a536d14d60af/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b285052e9cfd14cc452965839833e52aa2a82953597d8052e0e923e91c8c5579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b285052e9cfd14cc452965839833e52aa2a82953597d8052e0e923e91c8c5579/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8420a329aa507a59b761e8778e0c3ae8c04c20ea63ccc74d9328547862d9685/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8420a329aa507a59b761e8778e0c3ae8c04c20ea63ccc74d9328547862d9685/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dffa86620a6045e2442c2d70fc23b67884611501d81e8bc9760245dbd795e15e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dffa86620a6045e2442c2d70fc23b67884611501d81e8bc9760245dbd795e15e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2c19c519c02b28a65f483c6dccbbe9b22e3d77b809e98972034f7198ea71974/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2c19c519c02b28a65f483c6dccbbe9b22e3d77b809e98972034f7198ea71974/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fee12141e68d64c054e7de05847e6cb79216771ec54a0695882cef68ff72781b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fee12141e68d64c054e7de05847e6cb79216771ec54a0695882cef68ff72781b/userdata/shm major:0 minor:58 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00141100-9f3c-4e38-b3d5-04a31b4c76e3/volumes/kubernetes.io~projected/kube-api-access-48f7q:{mountpoint:/var/lib/kubelet/pods/00141100-9f3c-4e38-b3d5-04a31b4c76e3/volumes/kubernetes.io~projected/kube-api-access-48f7q major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~projected/kube-api-access-fs4wd:{mountpoint:/var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~projected/kube-api-access-fs4wd major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17db0bdd-c610-416d-b20e-b0f51a839205/volumes/kubernetes.io~projected/kube-api-access-bn96l:{mountpoint:/var/lib/kubelet/pods/17db0bdd-c610-416d-b20e-b0f51a839205/volumes/kubernetes.io~projected/kube-api-access-bn96l major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905/volumes/kubernetes.io~projected/kube-api-access-xhlmh:{mountpoint:/var/lib/kubelet/pods/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905/volumes/kubernetes.io~projected/kube-api-access-xhlmh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~projected/kube-api-access-tkggv:{mountpoint:/var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~projected/kube-api-access-tkggv major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/430e1cec-2732-4742-8eed-d2ba0fc4caf3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/430e1cec-2732-4742-8eed-d2ba0fc4caf3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51410712-fd08-4429-8d93-c0266660f335/volumes/kubernetes.io~projected/kube-api-access-f26td:{mountpoint:/var/lib/kubelet/pods/51410712-fd08-4429-8d93-c0266660f335/volumes/kubernetes.io~projected/kube-api-access-f26td major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~projected/kube-api-access-wbvbl:{mountpoint:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~projected/kube-api-access-wbvbl major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59e913f7-bc17-4001-bed8-348466e5966c/volumes/kubernetes.io~projected/kube-api-access-g6r65:{mountpoint:/var/lib/kubelet/pods/59e913f7-bc17-4001-bed8-348466e5966c/volumes/kubernetes.io~projected/kube-api-access-g6r65 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59e913f7-bc17-4001-bed8-348466e5966c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/59e913f7-bc17-4001-bed8-348466e5966c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b/volumes/kubernetes.io~projected/kube-api-access-9qrlf:{mountpoint:/var/lib/kubelet/pods/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b/volumes/kubernetes.io~projected/kube-api-access-9qrlf major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volumes/kubernetes.io~projected/kube-api-access-jqbnh:{mountpoint:/var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volumes/kubernetes.io~projected/kube-api-access-jqbnh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f/volumes/kubernetes.io~projected/kube-api-access-89zvn:{mountpoint:/var/lib/kubelet/pods/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f/volumes/kubernetes.io~projected/kube-api-access-89zvn major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~projected/kube-api-access-m69mv:{mountpoint:/var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~projected/kube-api-access-m69mv major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~projected/kube-api-access-xqqsg:{mountpoint:/var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~projected/kube-api-access-xqqsg major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88501d15-b241-4af1-af7a-eabc4c417a08/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/88501d15-b241-4af1-af7a-eabc4c417a08/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93cc76bb-77ed-493e-9207-c5bb8e30bc25/volumes/kubernetes.io~projected/kube-api-access-4n6tf:{mountpoint:/var/lib/kubelet/pods/93cc76bb-77ed-493e-9207-c5bb8e30bc25/volumes/kubernetes.io~projected/kube-api-access-4n6tf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93cc76bb-77ed-493e-9207-c5bb8e30bc25/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/93cc76bb-77ed-493e-9207-c5bb8e30bc25/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/alertmanager/9 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/config-reloader/6 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~empty-dir/config-out major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~projected/kube-api-access-zqhh9:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~projected/kube-api-access-zqhh9 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~projected/tls-assets major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/config-volume major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/web-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbcaff87-f298-42b7-8fce-0850c567d9c7/volumes/kubernetes.io~projected/kube-api-access-fh24b:{mountpoint:/var/lib/kubelet/pods/cbcaff87-f298-42b7-8fce-0850c567d9c7/volumes/kubernetes.io~projected/kube-api-access-fh24b major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~projected/kube-api-access-bg8hh:{mountpoint:/var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~projected/kube-api-access-bg8hh major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0ecce6a-ad31-4a68-9298-5f5618ece509/volumes/kubernetes.io~projected/kube-api-access-zhcfs:{mountpoint:/var/lib/kubelet/pods/e0ecce6a-ad31-4a68-9298-5f5618ece509/volumes/kubernetes.io~projected/kube-api-access-zhcfs major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9634e11-0c6c-4832-95a9-6665d679e1fd/volumes/kubernetes.io~projected/kube-api-access-fn4q2:{mountpoint:/var/lib/kubelet/pods/f9634e11-0c6c-4832-95a9-6665d679e1fd/volumes/kubernetes.io~projected/kube-api-access-fn4q2 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9634e11-0c6c-4832-95a9-6665d679e1fd/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/f9634e11-0c6c-4832-95a9-6665d679e1fd/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fffd3590-b7f0-41c9-b24f-1575b39c8f25/volumes/kubernetes.io~projected/kube-api-access-82fvz:{mountpoint:/var/lib/kubelet/pods/fffd3590-b7f0-41c9-b24f-1575b39c8f25/volumes/kubernetes.io~projected/kube-api-access-82fvz major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/00518aed21cb9ca5f7dfa432a331fa3137681622d206faa5ac87df39cd1ba3af/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/705b65fdc30043d30f2a37e2212343eef1ad6f14b178f6dd6711e2318ae69ae7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7ee8b36fa6ea5570ac70819089e601b266987fb83cac9056d1e57d4f61cee9b7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/edf092236348ec72e610720724c089c98be724d21ee1b9821bd94c427a32ab1a/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/754392a960d90825596567071a86a9d2104955669dfc99b584bc4a8f5ba2bb97/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/688f660316d9a79dbd90abc751ea1dd6316ffd90f35eb7ca2caf8e94eda0970e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7058c9f01b0ac1f2647ba84681860bbc24d5f210543c76379cf3a09a573b7046/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/fbf5740761167a7f2e528463b325715207190f6ce79f22ef5f42f26e2060a602/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/853d9ac13393eea932aedccecc271acc543a87b24892536b7ca04db946e2b8a0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/5be3f939541f55870f0afff80f1e4e26c9fa18bae7468ddf23ada933a8cc0cbd/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/864db79e20cf47af73f42668e0687c84a246395edf1f15d19ff27fb4ced7858a/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/e2268cbd50736989afb11f65a9b0003e9a49c0be6f185be7e9eee7f1c011de16/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/18d7ab06bc2317b9109a4c1a249325cad7bf95be3e8656a4856a039a1b72659e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9a61704e79ed9ddec1ed96aaa02b50175f3885fdc5a7a08988b5d8c1886f0586/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7486e4f15c5d8c583b039206c17329dc7fb8c7771c304a0e743af65112b02374/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d220ea1e4c727b2e09360a65b60084f03d24b02b448de9fd7578c5118a94b3d8/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5072178b2c22b4b88ddfc5a604311269cfddd8d75900bec5b696bdc543260236/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0c9c7a276e27406d7be0b5421714c358df1cb98478ed6cb25e32bc1b585b56a4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/386c4f060161a52a3ed0a96074249bf95e23b6ce087fc32de6f6b9b2852fdc54/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ac507b98a2f0a5b0e86d8a9614bad93498dcd2b1b118dc1b9479236675fc76e7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9398d963d6e8dc5565bc80eb0cf6499f80275264fb0e3ac3eb10fb358b015b39/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/af22e8d46df23426975a50ccc809905d2a72a84bbbe23bf9b13c00bfa8d82cf3/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/099b15696b2d8aa5a2e168f9e9a7fe43d155162e51843899a4ebb7312ed56dd8/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e6f46a3b961d69ff0f924b8539e812b9f433976b3e645a0548ddaa765c0375a5/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/188345bde22d75a8990e3d6cb9e870348ead19927d4d2359f290bcce2f45a03d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/4268d17d0ac6695e4780f7830eadb786f466cc8589a1db55e07a0af7614edc46/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/f8425eb479469653e5c891c346e66db872961228110b6bcc9b7c04a6025ef6ac/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/248e30826ac34ecc32e01647fd8046e66eb60a61de748ac07d1f604b065f15f0/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/949695c8283a157cebb5764790ceea2bf3d3b908e87ca247cceef2aa83f7b335/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/99f50568d82f0bf13d25c40a47848ef0a283b71ba089b808d09e5a7096c9f26a/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/3980aab1403a1903df47e8114e88a92ea05d30a8920e4ddce35999d5e7aefdeb/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/4d3bdd32087fc76eb288eaca313ce857a4f12003a6d8ab8e49b41baa749777f0/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/3b93bb102630db37de265281bbadd4152f648621ce15fff1f2bbbcf51499decb/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/3ec6e56d82de40e6960899d692ef8a535917e0dedefba9a04507162e13a0fe5a/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/6e668273cd7eeedc9525b04f1078b1dfe40bb428b9b989bc68004b7601cc2d12/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/642ac6c58fc9412063f9310f32552663352072ae17d42e32715f5603a1a08db2/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/44780e49a0bb2da1f8db1c85044887b9540484a4cd9f618d7cd8d07718226e8d/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/2a5ad42168bdbc068438d663c703848b2b396709281f4e6ffaa311e01d3e04bd/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/45669a97462c2dcdb3c61f0a668198c0bfb7f79213fe2292c0145beef41bda01/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/23b699d301dfb075ee905afe64dd815f1b62822b0cb8ceed5680a14f8046c6ae/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/724f74c260e5134ac3e9df74b721d54e21856bd22c4ad3467dc8fdd3f3c6481b/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/50933f05f2a45ccbbd6e2494e30a030501f4780c3535270b866e9183dbcd59b2/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/8dda87d1d82554f64878f71b7b489c6d3f6b3447910e909fb9ea18861a66fc5a/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/f47e7887db4d5b7695d9f7d7074b629ea58c4a366e4d817d2fce5b0d6fef34f3/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/a1e1b469bf660ae3c99425a76e640cbb1482c6a83f48dfa7ab5742f387d78bde/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/657e28995e44ba1f9b943426d1e42278a8fba2fb891aa3e5ed2feb7aed98621c/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/15619bc1a5ed299b082cfa6df7884a4c347043e783441138aa83750b11c2db89/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/5809175b158276a979afa1f9279570365dc4e303c09b0e346ff2ba64efa5358e/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/54e712304c7317b9526cf5680114831ecbda25142d3d45c0212c83e92cc785ed/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/38c5628f368106497f8901eba747d040d8a056e3db6e4bcab4ad677fe6aa905d/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/33027734e14ab4823a42783d0927f525a0a9cfa605a0de7439a4fefcf8ea172d/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/ad8c2f8b347a338f9e82fc57dc256c4ec259352445e51c4f9501cda3f616e20e/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/ea4d830b1801a511213392b65bb1cf88d5566f3c186e36f84062c7e630510c91/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/8f460ba6d8893f1a6d19dd778bb129ab5abab8152562d5128bbdf7f04c79a0b0/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/4d65217064361f872ed773727855352f533fb85f645c6c67666a11a0c4363fa4/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/c3f0926e0426322ab1098bef61e3be9acc9fb2e3743e911161419fd1212f347a/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/f50d415605d5b9be3584e27ada582ad0d815fc45a2226a232e2f1802652d4a38/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/1e454bfd9fb6293b39313e91cdd588eaa5d719eb7dde59c55b99d338404ebd03/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/0ff18cf9b890e87bfc7552f6840e2e5bd17cdb6b0e1c57258c23521102108af8/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/6213d4d3a2fda509cbfc75c05acb19ddb4844422ef6ddb6a6ae25b2cf1ccf92d/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/3e6b78f4ab88ec6fdf218aee8345389ba89a1e2971a487b1d9c98a1e4ec81f39/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/cb814efce6d4ab52037966e2961c597fbeba5540d4b84ff4e4c759caacc1b1c6/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/0f1bbf17082094fb41455fb1ea90d063d4dd5bd01d7b198950769a9183250f1d/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/887d39c6838ab618637aefd42eb8e52ba2de5f103783b5ab751ac8de04c46185/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/e6e27bfdea3d603945c489e908605fe9f2409c7d8d39cca4397b37bdaf16acd2/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/609f541b32eb107461c2982cd1bdd4c6b913d475f3295e144b0d36310e291ace/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/8b8044e574fd5ff74a778ca14646afb3d6d30ea16b6eed83ef324cee93ed4735/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3ee8fd8b741336ec26994b0313f3db0fe15b092312fe22c1eeff3ba7a8b95b86/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/5cb8d033cc927e5de3dc72c45c193bdc055120313808765c52118de68276aaba/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/5ba735bfb453b9ac9f676193a87b5ef0dad0f2f60b809e52932e77e39a520ec0/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/058d07b8d85216c1edd50e74512da313514e695a40002d1254e53843d364a630/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/589e60e19d1b1916d0a6710278a99af9f5eae0a8ffc752638c4cf4314d9ff3e5/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.175064    5393 manager.go:217] Machine: {Timestamp:2026-11-03 10:29:16.174541935 +0000 UTC m=+0.079396367 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099614 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26ce5bc24ffbf2c5dde0968cfa8d13 SystemUUID:ec26ce5b-c24f-fbf2-c5dd-e0968cfa8d13 BootID:e37a38c8-e62c-4636-8531-89a56b64e968 Filesystems:[{Device:/run/containers/storage/overlay-containers/53f0ab87129c8ba04c72fa95466ee09074e903d5635d7eaf2aa0840957ce8578/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~projected/kube-api-access-wbvbl DeviceMajor:0 DeviceMinor:264 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f/volumes/kubernetes.io~projected/kube-api-access-89zvn DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:244 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/829066a0ec2fa0b403b17eb3991f126d04587cbb459efb11898864b3b69059e6/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93cc76bb-77ed-493e-9207-c5bb8e30bc25/volumes/kubernetes.io~projected/kube-api-access-4n6tf DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~projected/kube-api-access-zqhh9 DeviceMajor:0 DeviceMinor:252 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:307 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dffa86620a6045e2442c2d70fc23b67884611501d81e8bc9760245dbd795e15e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:249 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbcaff87-f298-42b7-8fce-0850c567d9c7/volumes/kubernetes.io~projected/kube-api-access-fh24b DeviceMajor:0 DeviceMinor:257 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:259 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/136f55de77188ed0db84e1b9d019b11cd7cbacadff8cca77d8c3550722ff2335/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31f108a73bb5a733253a75aef7754adf64b5c60a911f34d0734038f5be172e33/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/51410712-fd08-4429-8d93-c0266660f335/volumes/kubernetes.io~projected/kube-api-access-f26td DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fee12141e68d64c054e7de05847e6cb79216771ec54a0695882cef68ff72781b/userdata/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~projected/kube-api-access-xqqsg DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:261 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:263 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b285052e9cfd14cc452965839833e52aa2a82953597d8052e0e923e91c8c5579/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2c19c519c02b28a65f483c6dccbbe9b22e3d77b809e98972034f7198ea71974/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:305 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/313968a59c031db374af80688503703d2d2d03ee89dfe0f05701b168f404de52/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:248 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59e913f7-bc17-4001-bed8-348466e5966c/volumes/kubernetes.io~projected/kube-api-access-g6r65 DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:258 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b/volumes/kubernetes.io~projected/kube-api-access-9qrlf DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fffd3590-b7f0-41c9-b24f-1575b39c8f25/volumes/kubernetes.io~projected/kube-api-access-82fvz DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:248 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/430e1cec-2732-4742-8eed-d2ba0fc4caf3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:256 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:266 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f5988790763ab7c1bc0394604de6b74d91b0c6c1ccc605043910d53b790bc80/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:247 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a43122507a4f5325f86d1f6ed224d4e78ca3e1c0233504474f90a536d14d60af/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00141100-9f3c-4e38-b3d5-04a31b4c76e3/volumes/kubernetes.io~projected/kube-api-access-48f7q DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59e913f7-bc17-4001-bed8-348466e5966c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:246 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:362 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:248 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:262 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88501d15-b241-4af1-af7a-eabc4c417a08/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/136a60959a855bfbeff212a5c5e618b6968c28f5b662533fc576518fe14e4e38/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volumes/kubernetes.io~projected/kube-api-access-jqbnh DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9634e11-0c6c-4832-95a9-6665d679e1fd/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:312 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91a30875268765bed1cc83c1bdc52cf8b049b9032f79a710d0ad0eeceeb86c5e/userdata/shm DeviceMajor:0 DeviceMinor:57 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:248 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905/volumes/kubernetes.io~projected/kube-api-access-xhlmh DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:243 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~projected/kube-api-access-m69mv DeviceMajor:0 DeviceMinor:363 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0ecce6a-ad31-4a68-9298-5f5618ece509/volumes/kubernetes.io~projected/kube-api-access-zhcfs DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2f8a0a98ec816ccc44f3d6ed26a5bd85ecc09db14a1226215d142e04c236257/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93cc76bb-77ed-493e-9207-c5bb8e30bc25/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~projected/kube-api-access-bg8hh DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9634e11-0c6c-4832-95a9-6665d679e1fd/volumes/kubernetes.io~projected/kube-api-access-fn4q2 DeviceMajor:0 DeviceMinor:313 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:358 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54b87353-bd2b-4a31-a5d3-9aa7582e0354/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:260 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b82d3e4f8156f97e9b04ebf383094b0e39d493e8ad2deacf3858e256a1c8889/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:245 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~projected/kube-api-access-fs4wd DeviceMajor:0 DeviceMinor:311 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:250 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1f00d4-439a-46b1-98fe-b595af207890/volumes/kubernetes.io~projected/kube-api-access-tkggv DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71f6bbd9-acd8-4534-8adb-ca7cd139d383/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e6497f39d5c52c7aa1e810c61867dbfb558b8077657bf04e020efb19643f4ae/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11b7246b-031f-46cd-ab96-814d2e5d4a72/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:306 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8420a329aa507a59b761e8778e0c3ae8c04c20ea63ccc74d9328547862d9685/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/161058e45d6663ee796adea96ae77c39d8bec7a3153b59883d19651862dc197c/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e70903a7e12534ad0b15d17570f2bea210477a91ac151b60bdd2cec61dc5286/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d16fcb39-bfc9-4d51-af89-14d83a3d07a1/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31eb62fa6614e04ced7dc4a8147116968a2b7343fa2d787f6c815248b9136fa1/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8115cbe0c0262d974bb9566912e76cc706913703b7a4bd2194018a99a7fb0b64/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:251 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3015f5c63a4e021ad2a0c057693a9cb293a9e8524a0a8561ef113da0017915ed/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/17db0bdd-c610-416d-b20e-b0f51a839205/volumes/kubernetes.io~projected/kube-api-access-bn96l DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1997dc5450e3cb7965eb3b05b42a223f7e84a2776fa84949036a56a23de384a2/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:250 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:136f55de77188ed MacAddress:06:98:da:91:01:7f Speed:10000 Mtu:8901} {Name:1997dc5450e3cb7 MacAddress:9e:f4:24:7d:b2:fd Speed:10000 Mtu:8901} {Name:3015f5c63a4e021 MacAddress:c6:17:8d:7d:f2:f6 Speed:10000 Mtu:8901} {Name:313968a59c031db MacAddress:76:79:1e:0a:c6:22 Speed:10000 Mtu:8901} {Name:31f108a73bb5a73 MacAddress:ee:6c:55:f4:3c:15 Speed:10000 Mtu:8901} {Name:3e70903a7e12534 MacAddress:2e:96:8d:f5:5e:75 Speed:10000 Mtu:8901} {Name:4e6497f39d5c52c MacAddress:ae:8f:d2:09:22:9e Speed:10000 Mtu:8901} {Name:6f5988790763ab7 MacAddress:d6:09:cc:7a:25:db Speed:10000 Mtu:8901} {Name:829066a0ec2fa0b MacAddress:da:fd:79:0a:21:96 Speed:10000 Mtu:8901} {Name:a43122507a4f532 MacAddress:2a:34:7d:96:8c:74 Speed:10000 Mtu:8901} {Name:b285052e9cfd14c MacAddress:0a:06:33:5b:bb:74 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c3:8b:74:71 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:d8:1a:1c:0d:ce Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:c3:8b:74:71 Speed:0 Mtu:9001} {Name:f2c19c519c02b28 MacAddress:92:ed:91:81:b4:d8 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:aa:2e:41:62:63:0c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:b6:b9:7e:87:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.175471    5393 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.175517    5393 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.175843    5393 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.175867    5393 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-12.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176030    5393 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176038    5393 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176048    5393 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176069    5393 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176157    5393 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176334    5393 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176457    5393 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176493    5393 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176503    5393 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176513    5393 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176523    5393 kubelet.go:405] "Adding apiserver pod source"
Nov 03 10:29:16.176548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.176531    5393 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 10:29:16.177280 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.177213    5393 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 10:29:16.177280 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.177222    5393 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 10:29:16.178126 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.178096    5393 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 10:29:16.181845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.181827    5393 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 10:29:16.181845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.181841    5393 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 10:29:16.181972 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.181963    5393 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 10:29:16.182163 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182151    5393 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 10:29:16.182204 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182186    5393 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 10:29:16.182301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182291    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182305    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182312    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182318    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182324    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182330    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182336    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182341    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182347    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182353    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182361    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 10:29:16.182367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182370    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 10:29:16.182862 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182395    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 10:29:16.182862 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.182404    5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 10:29:16.186025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.186012    5393 server.go:1263] "Started kubelet"
Nov 03 10:29:16.186439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.186112    5393 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 10:29:16.186439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.186142    5393 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 10:29:16.186439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.186227    5393 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 10:29:16.186613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.186535    5393 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 10:29:16.186645 ip-10-0-134-12 systemd[1]: Started Kubernetes Kubelet.
Nov 03 10:29:16.187360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.187343    5393 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 10:29:16.190438 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.190419    5393 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 10:29:16.190529 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.190444    5393 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 10:29:16.190601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.190538    5393 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 10:29:16.190601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.190557    5393 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 10:23:36 +0000 UTC" deadline="2028-05-25 14:05:44.335559206 +0000 UTC"
Nov 03 10:29:16.190601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.190580    5393 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 10:29:16.190601 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:16.190546    5393 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-12.us-east-2.compute.internal\" not found"
Nov 03 10:29:16.190601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.190579    5393 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13659h36m28.144983069s"
Nov 03 10:29:16.190601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.190568    5393 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 10:29:16.192441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.192361    5393 factory.go:55] Registering systemd factory
Nov 03 10:29:16.193552 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.193476    5393 factory.go:223] Registration of the systemd container factory successfully
Nov 03 10:29:16.196111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.196079    5393 factory.go:153] Registering CRI-O factory
Nov 03 10:29:16.196111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.196107    5393 factory.go:223] Registration of the crio container factory successfully
Nov 03 10:29:16.196246 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.196211    5393 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 10:29:16.196297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.196251    5393 factory.go:103] Registering Raw factory
Nov 03 10:29:16.196297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.196272    5393 manager.go:1199] Started watching for new ooms in manager
Nov 03 10:29:16.196981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.196945    5393 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:29:16.196981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.196965    5393 manager.go:319] Starting recovery of all containers
Nov 03 10:29:16.197117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197059    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11b7246b-031f-46cd-ab96-814d2e5d4a72" volumeName="kubernetes.io/empty-dir/11b7246b-031f-46cd-ab96-814d2e5d4a72-audit-log" seLinuxMountContext=""
Nov 03 10:29:16.197198 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197120    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51410712-fd08-4429-8d93-c0266660f335" volumeName="kubernetes.io/empty-dir/51410712-fd08-4429-8d93-c0266660f335-tmp-dir" seLinuxMountContext=""
Nov 03 10:29:16.197234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197205    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a6abe6-d585-4c92-9ee8-2707962af39b" volumeName="kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config" seLinuxMountContext=""
Nov 03 10:29:16.197295 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197279    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cf9d6c-4dc2-49e0-93c9-d6115eeeb905" volumeName="kubernetes.io/configmap/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-config-volume" seLinuxMountContext=""
Nov 03 10:29:16.197347 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197302    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="430e1cec-2732-4742-8eed-d2ba0fc4caf3" volumeName="kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret" seLinuxMountContext=""
Nov 03 10:29:16.197347 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197321    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856fad64-8c12-4623-b444-5d99425a7505" volumeName="kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv" seLinuxMountContext=""
Nov 03 10:29:16.197409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197377    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93cc76bb-77ed-493e-9207-c5bb8e30bc25" volumeName="kubernetes.io/secret/93cc76bb-77ed-493e-9207-c5bb8e30bc25-cert" seLinuxMountContext=""
Nov 03 10:29:16.197439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197426    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fcb39-bfc9-4d51-af89-14d83a3d07a1" volumeName="kubernetes.io/projected/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-kube-api-access-bg8hh" seLinuxMountContext=""
Nov 03 10:29:16.197466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197458    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fcb39-bfc9-4d51-af89-14d83a3d07a1" volumeName="kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 10:29:16.197526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197490    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7532f5ee-e2ed-40a2-9d9e-277a392e1c9f" volumeName="kubernetes.io/projected/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-kube-api-access-89zvn" seLinuxMountContext=""
Nov 03 10:29:16.197560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197537    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a6abe6-d585-4c92-9ee8-2707962af39b" volumeName="kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config" seLinuxMountContext=""
Nov 03 10:29:16.197560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197552    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume" seLinuxMountContext=""
Nov 03 10:29:16.197611 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197582    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-grpc-tls" seLinuxMountContext=""
Nov 03 10:29:16.197642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197605    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cf9d6c-4dc2-49e0-93c9-d6115eeeb905" volumeName="kubernetes.io/projected/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-kube-api-access-xhlmh" seLinuxMountContext=""
Nov 03 10:29:16.197642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197628    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9" seLinuxMountContext=""
Nov 03 10:29:16.197759 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197666    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 10:29:16.197759 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197693    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9634e11-0c6c-4832-95a9-6665d679e1fd" volumeName="kubernetes.io/secret/f9634e11-0c6c-4832-95a9-6665d679e1fd-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 10:29:16.197814 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197764    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00141100-9f3c-4e38-b3d5-04a31b4c76e3" volumeName="kubernetes.io/projected/00141100-9f3c-4e38-b3d5-04a31b4c76e3-kube-api-access-48f7q" seLinuxMountContext=""
Nov 03 10:29:16.197849 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197832    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/projected/54b87353-bd2b-4a31-a5d3-9aa7582e0354-kube-api-access-wbvbl" seLinuxMountContext=""
Nov 03 10:29:16.197892 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197850    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 10:29:16.197892 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197877    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f6bbd9-acd8-4534-8adb-ca7cd139d383" volumeName="kubernetes.io/secret/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 10:29:16.197983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197909    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856fad64-8c12-4623-b444-5d99425a7505" volumeName="kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp" seLinuxMountContext=""
Nov 03 10:29:16.197983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197950    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88501d15-b241-4af1-af7a-eabc4c417a08" volumeName="kubernetes.io/configmap/88501d15-b241-4af1-af7a-eabc4c417a08-konnectivity-ca" seLinuxMountContext=""
Nov 03 10:29:16.197983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.197964    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 10:29:16.199400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199367    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b" volumeName="kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs" seLinuxMountContext=""
Nov 03 10:29:16.199400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199397    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f6bbd9-acd8-4534-8adb-ca7cd139d383" volumeName="kubernetes.io/projected/71f6bbd9-acd8-4534-8adb-ca7cd139d383-kube-api-access-jqbnh" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199409    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856fad64-8c12-4623-b444-5d99425a7505" volumeName="kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199421    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856fad64-8c12-4623-b444-5d99425a7505" volumeName="kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199440    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fcb39-bfc9-4d51-af89-14d83a3d07a1" volumeName="kubernetes.io/empty-dir/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-textfile" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199453    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a6abe6-d585-4c92-9ee8-2707962af39b" volumeName="kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199467    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11b7246b-031f-46cd-ab96-814d2e5d4a72" volumeName="kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-client-ca-bundle" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199483    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f6bbd9-acd8-4534-8adb-ca7cd139d383" volumeName="kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199499    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856fad64-8c12-4623-b444-5d99425a7505" volumeName="kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199514    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88501d15-b241-4af1-af7a-eabc4c417a08" volumeName="kubernetes.io/secret/88501d15-b241-4af1-af7a-eabc4c417a08-agent-certs" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199543    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93cc76bb-77ed-493e-9207-c5bb8e30bc25" volumeName="kubernetes.io/projected/93cc76bb-77ed-493e-9207-c5bb8e30bc25-kube-api-access-4n6tf" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199559    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f6bbd9-acd8-4534-8adb-ca7cd139d383" volumeName="kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-config" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199575    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1f00d4-439a-46b1-98fe-b595af207890" volumeName="kubernetes.io/projected/2a1f00d4-439a-46b1-98fe-b595af207890-kube-api-access-tkggv" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199589    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11b7246b-031f-46cd-ab96-814d2e5d4a72" volumeName="kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199607    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11b7246b-031f-46cd-ab96-814d2e5d4a72" volumeName="kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 10:29:16.199615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199620    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cf9d6c-4dc2-49e0-93c9-d6115eeeb905" volumeName="kubernetes.io/secret/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-metrics-tls" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199633    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199645    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199658    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199671    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199686    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e913f7-bc17-4001-bed8-348466e5966c" volumeName="kubernetes.io/configmap/59e913f7-bc17-4001-bed8-348466e5966c-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199699    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a6abe6-d585-4c92-9ee8-2707962af39b" volumeName="kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199710    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199723    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbcaff87-f298-42b7-8fce-0850c567d9c7" volumeName="kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199735    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ecce6a-ad31-4a68-9298-5f5618ece509" volumeName="kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-cni-binary-copy" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199748    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9634e11-0c6c-4832-95a9-6665d679e1fd" volumeName="kubernetes.io/projected/f9634e11-0c6c-4832-95a9-6665d679e1fd-kube-api-access-fn4q2" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199780    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fffd3590-b7f0-41c9-b24f-1575b39c8f25" volumeName="kubernetes.io/projected/fffd3590-b7f0-41c9-b24f-1575b39c8f25-kube-api-access-82fvz" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199793    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00141100-9f3c-4e38-b3d5-04a31b4c76e3" volumeName="kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-binary-copy" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199806    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199821    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a6abe6-d585-4c92-9ee8-2707962af39b" volumeName="kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199851    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fcb39-bfc9-4d51-af89-14d83a3d07a1" volumeName="kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-metrics-client-ca" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199863    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fffd3590-b7f0-41c9-b24f-1575b39c8f25" volumeName="kubernetes.io/configmap/fffd3590-b7f0-41c9-b24f-1575b39c8f25-iptables-alerter-script" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199888    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00141100-9f3c-4e38-b3d5-04a31b4c76e3" volumeName="kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199906    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199941    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f6bbd9-acd8-4534-8adb-ca7cd139d383" volumeName="kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-env-overrides" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199959    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7532f5ee-e2ed-40a2-9d9e-277a392e1c9f" volumeName="kubernetes.io/configmap/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-serviceca" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199972    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856fad64-8c12-4623-b444-5d99425a7505" volumeName="kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199985    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.199999    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200030    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ecce6a-ad31-4a68-9298-5f5618ece509" volumeName="kubernetes.io/projected/e0ecce6a-ad31-4a68-9298-5f5618ece509-kube-api-access-zhcfs" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200045    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11b7246b-031f-46cd-ab96-814d2e5d4a72" volumeName="kubernetes.io/projected/11b7246b-031f-46cd-ab96-814d2e5d4a72-kube-api-access-fs4wd" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200060    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856fad64-8c12-4623-b444-5d99425a7505" volumeName="kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200076    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200090    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11b7246b-031f-46cd-ab96-814d2e5d4a72" volumeName="kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200104    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51410712-fd08-4429-8d93-c0266660f335" volumeName="kubernetes.io/projected/51410712-fd08-4429-8d93-c0266660f335-kube-api-access-f26td" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200118    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/empty-dir/54b87353-bd2b-4a31-a5d3-9aa7582e0354-tmp" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200131    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e913f7-bc17-4001-bed8-348466e5966c" volumeName="kubernetes.io/secret/59e913f7-bc17-4001-bed8-348466e5966c-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200143    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b" volumeName="kubernetes.io/projected/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-kube-api-access-9qrlf" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200157    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200170    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200191    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1f00d4-439a-46b1-98fe-b595af207890" volumeName="kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-etc-tuned" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200204    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1f00d4-439a-46b1-98fe-b595af207890" volumeName="kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-tmp" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200217    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fcb39-bfc9-4d51-af89-14d83a3d07a1" volumeName="kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-tls" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200223    5393 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200230    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ecce6a-ad31-4a68-9298-5f5618ece509" volumeName="kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-daemon-config" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200244    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17db0bdd-c610-416d-b20e-b0f51a839205" volumeName="kubernetes.io/projected/17db0bdd-c610-416d-b20e-b0f51a839205-kube-api-access-bn96l" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200258    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e913f7-bc17-4001-bed8-348466e5966c" volumeName="kubernetes.io/empty-dir/59e913f7-bc17-4001-bed8-348466e5966c-data-volume" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200274    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e913f7-bc17-4001-bed8-348466e5966c" volumeName="kubernetes.io/projected/59e913f7-bc17-4001-bed8-348466e5966c-kube-api-access-g6r65" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200287    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="856fad64-8c12-4623-b444-5d99425a7505" volumeName="kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200301    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a6abe6-d585-4c92-9ee8-2707962af39b" volumeName="kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200314    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" volumeName="kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db" seLinuxMountContext=""
Nov 03 10:29:16.200305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200327    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16fcb39-bfc9-4d51-af89-14d83a3d07a1" volumeName="kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 10:29:16.203234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200342    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a6abe6-d585-4c92-9ee8-2707962af39b" volumeName="kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca" seLinuxMountContext=""
Nov 03 10:29:16.203234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200357    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b87353-bd2b-4a31-a5d3-9aa7582e0354" volumeName="kubernetes.io/configmap/54b87353-bd2b-4a31-a5d3-9aa7582e0354-metrics-client-ca" seLinuxMountContext=""
Nov 03 10:29:16.203234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200372    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00141100-9f3c-4e38-b3d5-04a31b4c76e3" volumeName="kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 10:29:16.203234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200385    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11b7246b-031f-46cd-ab96-814d2e5d4a72" volumeName="kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 10:29:16.203234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200400    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cf9d6c-4dc2-49e0-93c9-d6115eeeb905" volumeName="kubernetes.io/empty-dir/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-tmp-dir" seLinuxMountContext=""
Nov 03 10:29:16.203234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200413    5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a6abe6-d585-4c92-9ee8-2707962af39b" volumeName="kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg" seLinuxMountContext=""
Nov 03 10:29:16.203234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200425    5393 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 10:29:16.203234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.200436    5393 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 10:29:16.203741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.203722    5393 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:29:16.204268 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.204248    5393 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 10:29:16.220819 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.220766    5393 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 10:29:16.222267 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.222222    5393 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 10:29:16.222267 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.222248    5393 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 10:29:16.222267 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.222271    5393 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 10:29:16.222515 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:16.222313    5393 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 10:29:16.225904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.225875    5393 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 10:29:16.236224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.236195    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:29:16.236451 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.236278    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:29:16.241677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.241645    5393 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="6765bb4cfc8f203cd18dc22572a18026eb8c7156b2d2ddd686acb57ebb168a52" exitCode=0
Nov 03 10:29:16.241677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.241675    5393 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="70715e5516c3f005d69670851492704b14e33d98d8691289fbf75d9310944386" exitCode=0
Nov 03 10:29:16.241823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.241691    5393 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="0d4c25b46974f6020bea1a91a84b4305c6ce106d5c818357dc7b650be5347c5b" exitCode=0
Nov 03 10:29:16.241823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.241699    5393 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="7cd48c855586ef5d29bd0e1f73010966da654d2e74e014e34473f3d6eb8e12d9" exitCode=0
Nov 03 10:29:16.241823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.241707    5393 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="d38a2ab9a83266c7548888e0773d1f3ee5d29375755784e14e7570e940e119ac" exitCode=0
Nov 03 10:29:16.241823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.241715    5393 generic.go:356] "Generic (PLEG): container finished" podID="00141100-9f3c-4e38-b3d5-04a31b4c76e3" containerID="b811939b1a45d8544dfdff6c58def701438fedce6ba63eace3a2e01669ce3ed5" exitCode=0
Nov 03 10:29:16.258375 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.258348    5393 generic.go:356] "Generic (PLEG): container finished" podID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerID="1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c" exitCode=0
Nov 03 10:29:16.262041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.262019    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:29:16.262456 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.262428    5393 generic.go:356] "Generic (PLEG): container finished" podID="71f6bbd9-acd8-4534-8adb-ca7cd139d383" containerID="137fae92e957983988ff501237fd5241b2af6a217a9fbad09542d09308618d26" exitCode=1
Nov 03 10:29:16.274102 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.274068    5393 generic.go:356] "Generic (PLEG): container finished" podID="d16fcb39-bfc9-4d51-af89-14d83a3d07a1" containerID="b7509b55c8c2b1e21c0d5e92cab3fcfd60a6badce3ab9f8be10fcd867cf4a9fb" exitCode=0
Nov 03 10:29:16.279052 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.279028    5393 generic.go:356] "Generic (PLEG): container finished" podID="f179e0cd67151dd9aa7b9a87412922fb" containerID="cb2da0cf3383e14be726d9d44071fb9bc9ec6299465cfe5c7449158ba2f6afb2" exitCode=0
Nov 03 10:29:16.306879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.306856    5393 manager.go:324] Recovery completed
Nov 03 10:29:16.322671 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:16.322635    5393 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 10:29:16.329479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329454    5393 cpu_manager.go:226] "Starting" policy="none"
Nov 03 10:29:16.329479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329469    5393 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 10:29:16.329479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329485    5393 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 10:29:16.329722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329605    5393 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 10:29:16.329722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329615    5393 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 10:29:16.329722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329632    5393 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 10:29:16.329722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329637    5393 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 10:29:16.329722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329642    5393 policy_none.go:50] "Start"
Nov 03 10:29:16.329722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329651    5393 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 10:29:16.329722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329659    5393 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 10:29:16.329910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329825    5393 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 10:29:16.329910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329833    5393 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 10:29:16.329910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.329838    5393 policy_none.go:44] "Start"
Nov 03 10:29:16.336878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.336847    5393 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 10:29:16.341597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.341584    5393 manager.go:341] "Starting Device Plugin manager"
Nov 03 10:29:16.342100 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:16.341611    5393 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 10:29:16.342100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.342100    5393 server.go:87] "Starting device plugin registration server"
Nov 03 10:29:16.342355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.342344    5393 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 10:29:16.342403 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.342355    5393 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 10:29:16.342468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.342449    5393 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 10:29:16.342581 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.342548    5393 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 10:29:16.342644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.342591    5393 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 10:29:16.342644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.342597    5393 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 10:29:16.342741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.342648    5393 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T10:29:16.342579237Z","UUID":"a233b470-7c09-4755-960e-9795afbf4e77","Handler":null,"Name":"","Endpoint":""}
Nov 03 10:29:16.344051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.344026    5393 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 10:29:16.344051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.344054    5393 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 10:29:16.344291 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:16.344230    5393 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 10:29:16.443157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.443066    5393 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 10:29:16.444413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.444394    5393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 10:29:16.444521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.444429    5393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 10:29:16.444521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.444444    5393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 10:29:16.444521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.444506    5393 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.457442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.457418    5393 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.457513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.457481    5393 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.523808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.523755    5393 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"]
Nov 03 10:29:16.524471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.524457    5393 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.524568 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.524434    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"f179e0cd67151dd9aa7b9a87412922fb","Type":"ContainerStarted","Data":"f2a166e97a08fbe0238171c622cd915060b31331d3a730db48a62b9a97a94aef"}
Nov 03 10:29:16.524601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.524579    5393 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.524601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.524589    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"f179e0cd67151dd9aa7b9a87412922fb","Type":"ContainerDied","Data":"cb2da0cf3383e14be726d9d44071fb9bc9ec6299465cfe5c7449158ba2f6afb2"}
Nov 03 10:29:16.524685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.524614    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"f179e0cd67151dd9aa7b9a87412922fb","Type":"ContainerStarted","Data":"91a30875268765bed1cc83c1bdc52cf8b049b9032f79a710d0ad0eeceeb86c5e"}
Nov 03 10:29:16.524685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.524631    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"3041328657ebd4fdb2a0f070aa3e29b6","Type":"ContainerStarted","Data":"2ce1d4a71baf36d12295b12d6be36d7fd9b005fb2de2e785de986f7c006a624a"}
Nov 03 10:29:16.524685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.524650    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal" event={"ID":"3041328657ebd4fdb2a0f070aa3e29b6","Type":"ContainerStarted","Data":"fee12141e68d64c054e7de05847e6cb79216771ec54a0695882cef68ff72781b"}
Nov 03 10:29:16.537157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.537138    5393 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:29:16.537259 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:16.537181    5393 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.540140 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.540128    5393 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:29:16.540184 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:16.540163    5393 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.606571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.606538    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.606571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.606573    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.606741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.606590    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3041328657ebd4fdb2a0f070aa3e29b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"3041328657ebd4fdb2a0f070aa3e29b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.707231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.707146    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.707231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.707186    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3041328657ebd4fdb2a0f070aa3e29b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"3041328657ebd4fdb2a0f070aa3e29b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.707231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.707211    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.707430 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.707261    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.707430 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.707265    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3041328657ebd4fdb2a0f070aa3e29b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"3041328657ebd4fdb2a0f070aa3e29b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:16.707430 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:16.707260    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f179e0cd67151dd9aa7b9a87412922fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" (UID: \"f179e0cd67151dd9aa7b9a87412922fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:17.177135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.177102    5393 apiserver.go:52] "Watching apiserver"
Nov 03 10:29:17.193733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.193713    5393 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 10:29:17.194315 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.194295    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-576b7767c8-btzbj","openshift-monitoring/alertmanager-main-0","openshift-network-operator/iptables-alerter-9ft6m","openshift-ovn-kubernetes/ovnkube-node-6t57h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9","openshift-monitoring/node-exporter-78k5n","openshift-console/console-59787cccbc-9wk85","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal","kube-system/konnectivity-agent-9d2nl","openshift-monitoring/thanos-querier-76b7876cf5-7mrjf","openshift-multus/multus-j7kpq","kube-system/global-pull-secret-syncer-w4txd","openshift-cluster-node-tuning-operator/tuned-vgs7g","openshift-monitoring/monitoring-plugin-77f4f76594-ttks6","openshift-multus/multus-additional-cni-plugins-nb997","openshift-dns/dns-default-9zqgx","openshift-ingress-canary/ingress-canary-kjzdq","openshift-monitoring/metrics-server-767d4fbf8c-86qwm","kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal","openshift-image-registry/node-ca-67pqq","openshift-insights/insights-runtime-extractor-98krd","openshift-dns/node-resolver-crz98","openshift-multus/network-metrics-daemon-wm4x2","openshift-network-diagnostics/network-check-target-cd6vm"]
Nov 03 10:29:17.204665 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.204636    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.204821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.204798    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.204888 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.204831    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.204888 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.204846    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.204888 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.204835    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.204888 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.204878    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.205096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.204948    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 10:29:17.205096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.204952    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.205096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.205002    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.205096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.205018    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.205096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.205034    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.205799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.205764    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ssvsh\""
Nov 03 10:29:17.205799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.205786    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 10:29:17.207065 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207043    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 10:29:17.207172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207085    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 10:29:17.207172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207104    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lq8vs\""
Nov 03 10:29:17.207172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207134    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zcr5r\""
Nov 03 10:29:17.207172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207166    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nxhw\""
Nov 03 10:29:17.207409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207235    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 10:29:17.207409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207271    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xjpbn\""
Nov 03 10:29:17.207409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207302    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-trm8g\""
Nov 03 10:29:17.207409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207336    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 10:29:17.207409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207360    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 10:29:17.207409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207374    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 10:29:17.207409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207393    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w9j8\""
Nov 03 10:29:17.207409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207399    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jkrmv\""
Nov 03 10:29:17.207794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207415    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.207794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207429    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 10:29:17.207794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.207450    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.208714 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208692    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 10:29:17.208804 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208739    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.208857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208838    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 10:29:17.208948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208891    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.208999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208947    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 10:29:17.208999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208971    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.208999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208982    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 10:29:17.208999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208992    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5hs5j\""
Nov 03 10:29:17.208999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.208997    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 10:29:17.209225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.209012    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pthxs\""
Nov 03 10:29:17.209225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.209033    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 10:29:17.209225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.209039    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-khtn4\""
Nov 03 10:29:17.209225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.209055    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 10:29:17.209225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.209062    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.209225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.209066    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4c7q2\""
Nov 03 10:29:17.209225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.209078    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 10:29:17.209225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.209088    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wbpkj\""
Nov 03 10:29:17.292715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.292683    5393 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:17.292853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.292792    5393 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:17.293752 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.293729    5393 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 10:29:17.303948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.303491    5393 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:29:17.304082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.304007    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.304169 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.304153    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.304608 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.304590    5393 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 10:29:17.304758 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:17.304741    5393 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:17.305199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.305178    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 10:29:17.306193 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.306174    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.306418 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.306403    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 10:29:17.306579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.306565    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 10:29:17.306693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.306676    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.306970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.306954    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 10:29:17.307122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.307108    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 10:29:17.307531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.307513    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.307816 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.307800    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.308497 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.308389    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 10:29:17.308497 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.308432    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 10:29:17.308497 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.308459    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 10:29:17.308682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.308521    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 10:29:17.308682 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:17.308542    5393 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal"
Nov 03 10:29:17.308682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.308610    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 10:29:17.308682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.308661    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 10:29:17.309218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.308983    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 10:29:17.309218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309076    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 10:29:17.309218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309131    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 10:29:17.309218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309217    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 10:29:17.309559 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309538    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5cc62\""
Nov 03 10:29:17.309726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309707    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 10:29:17.309797 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309777    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w9nrb\""
Nov 03 10:29:17.309844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309828    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjjb4\""
Nov 03 10:29:17.309888 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309860    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 10:29:17.309987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.309972    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 10:29:17.310079 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.310029    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fnqcd\""
Nov 03 10:29:17.310189 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.310159    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 10:29:17.310348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.310321    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 10:29:17.310749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.310730    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 10:29:17.311069 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.311055    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 10:29:17.311152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.311074    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 10:29:17.311152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.311124    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gqtlm\""
Nov 03 10:29:17.311152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.311137    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 10:29:17.311309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.311163    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 10:29:17.311309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.311162    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 10:29:17.311309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.311188    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8520rab7os4sv\""
Nov 03 10:29:17.311309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.311243    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 10:29:17.312488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312407    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 10:29:17.312488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312476    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.312655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312509    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cnibin\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.312655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312535    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-k8s-cni-cncf-io\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.312655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312565    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.312655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312627    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-etc-tuned\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.312843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312702    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-hostroot\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.312843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312731    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.312843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312748    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-etc-tuned\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.312843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312791    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.312843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312817    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-cni-binary-copy\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.313105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.312883    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.313105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313021    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-bin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.313105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313056    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.313105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313057    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.313105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313093    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-host\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:29:17.313351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313118    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-metrics-client-ca\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.313351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313153    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.313351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313204    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-cni-binary-copy\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.313351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313260    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysconfig\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.313351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313279    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.313351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313300    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-lib-modules\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.313351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313334    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-metrics-client-ca\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.313351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313347    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-host\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313366    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313374    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313414    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313574    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-ovn\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313577    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-metrics-certs\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313619    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313636    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313649    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313676    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-sys\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.313726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313710    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313744    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-sys-fs\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313770    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zvn\" (UniqueName: \"kubernetes.io/projected/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-kube-api-access-89zvn\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313796    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-etc-kubernetes\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313821    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313847    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-client-certs\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313883    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-netd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313909    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbvbl\" (UniqueName: \"kubernetes.io/projected/54b87353-bd2b-4a31-a5d3-9aa7582e0354-kube-api-access-wbvbl\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313946    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430e1cec-2732-4742-8eed-d2ba0fc4caf3-original-pull-secret\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313955    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-systemd\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313940    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.313980    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-multus\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314008    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e913f7-bc17-4001-bed8-348466e5966c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314034    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314060    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qrlf\" (UniqueName: \"kubernetes.io/projected/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-kube-api-access-9qrlf\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314073    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7whcj\""
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314090    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-kubelet-config\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314115    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-system-cni-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314129    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c9gx6\""
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314145    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e913f7-bc17-4001-bed8-348466e5966c-crio-socket\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314174    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-serviceca\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314175    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-client-certs\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314205    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 10:29:17.314211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314213    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-dbus\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314263    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314440    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314517    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314542    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e913f7-bc17-4001-bed8-348466e5966c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314637    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-slash\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314683    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovn-node-metrics-cert\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314829    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314899    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314962    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlmh\" (UniqueName: \"kubernetes.io/projected/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-kube-api-access-xhlmh\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.314985    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315026    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315051    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-modprobe-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315073    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-cnibin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315103    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.315245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315111    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315271    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-root\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315323    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-config-volume\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315375    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315445    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-socket-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315491    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315563    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-metrics-server-audit-profiles\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315590    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-systemd-units\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315625    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-node-log\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315658    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315685    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315712    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqsg\" (UniqueName: \"kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315732    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 10:29:17.315769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315737    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315802    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315832    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-wtmp\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315844    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315882    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93cc76bb-77ed-493e-9207-c5bb8e30bc25-cert\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315893    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315909    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4wd\" (UniqueName: \"kubernetes.io/projected/11b7246b-031f-46cd-ab96-814d2e5d4a72-kube-api-access-fs4wd\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315964    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.315990    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-metrics-tls\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316015    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fffd3590-b7f0-41c9-b24f-1575b39c8f25-iptables-alerter-script\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316042    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcfs\" (UniqueName: \"kubernetes.io/projected/e0ecce6a-ad31-4a68-9298-5f5618ece509-kube-api-access-zhcfs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316066    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-textfile\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316091    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316116    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4q2\" (UniqueName: \"kubernetes.io/projected/f9634e11-0c6c-4832-95a9-6665d679e1fd-kube-api-access-fn4q2\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316119    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316225    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-script-lib\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316244    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316263    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-netns\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316279    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6r65\" (UniqueName: \"kubernetes.io/projected/59e913f7-bc17-4001-bed8-348466e5966c-kube-api-access-g6r65\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316282    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-serviceca\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316296    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316299    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316312    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn96l\" (UniqueName: \"kubernetes.io/projected/17db0bdd-c610-416d-b20e-b0f51a839205-kube-api-access-bn96l\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316329    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8hh\" (UniqueName: \"kubernetes.io/projected/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-kube-api-access-bg8hh\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316345    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51410712-fd08-4429-8d93-c0266660f335-tmp-dir\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316375    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316388    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b87353-bd2b-4a31-a5d3-9aa7582e0354-tmp\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.316397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316403    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-kubernetes\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316424    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkggv\" (UniqueName: \"kubernetes.io/projected/2a1f00d4-439a-46b1-98fe-b595af207890-kube-api-access-tkggv\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316448    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316455    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fffd3590-b7f0-41c9-b24f-1575b39c8f25-iptables-alerter-script\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316473    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316475    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-config-volume\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316489    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316665    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316747    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316838    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93cc76bb-77ed-493e-9207-c5bb8e30bc25-cert\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316863    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316878    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.316974    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54b87353-bd2b-4a31-a5d3-9aa7582e0354-tmp\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317065    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-metrics-tls\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317114    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-script-lib\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317186    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317238    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-textfile\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317307    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-metrics-server-audit-profiles\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317349    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317351    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51410712-fd08-4429-8d93-c0266660f335-tmp-dir\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317371    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317371    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317379    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317405    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-conf\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317404    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-30l0kuump9kjj\""
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317441    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-kubelet\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317457    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qpnvs\""
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317481    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317501    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-kubelet\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317545    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-netns\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317643    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-sys\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.317713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317654    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317737    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fvz\" (UniqueName: \"kubernetes.io/projected/fffd3590-b7f0-41c9-b24f-1575b39c8f25-kube-api-access-82fvz\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317770    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317819    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-daemon-config\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317846    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.317883    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovn-node-metrics-cert\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318029    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69mv\" (UniqueName: \"kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318090    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6tf\" (UniqueName: \"kubernetes.io/projected/93cc76bb-77ed-493e-9207-c5bb8e30bc25-kube-api-access-4n6tf\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318124    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88501d15-b241-4af1-af7a-eabc4c417a08-agent-certs\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318156    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318192    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-run\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318245    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-conf-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318247    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-daemon-config\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318275    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318304    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318322    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318448    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88501d15-b241-4af1-af7a-eabc4c417a08-agent-certs\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318468    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-socket-dir-parent\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318500    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318537    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318578    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e913f7-bc17-4001-bed8-348466e5966c-data-volume\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318612    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e913f7-bc17-4001-bed8-348466e5966c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318650    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318691    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e913f7-bc17-4001-bed8-348466e5966c-data-volume\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318713    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318780    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48f7q\" (UniqueName: \"kubernetes.io/projected/00141100-9f3c-4e38-b3d5-04a31b4c76e3-kube-api-access-48f7q\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318826    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318845    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-system-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318852    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e913f7-bc17-4001-bed8-348466e5966c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318862    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318884    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-bin\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318900    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318951    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318968    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51410712-fd08-4429-8d93-c0266660f335-hosts-file\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.318983    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88501d15-b241-4af1-af7a-eabc4c417a08-konnectivity-ca\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319078    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319103    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319121    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9634e11-0c6c-4832-95a9-6665d679e1fd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319136    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/11b7246b-031f-46cd-ab96-814d2e5d4a72-audit-log\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319151    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-client-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319167    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-var-lib-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319183    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-multus-certs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319195    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319228    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-registration-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319250    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-device-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319274    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fffd3590-b7f0-41c9-b24f-1575b39c8f25-host-slash\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319298    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-env-overrides\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319323    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-grpc-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.319311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319343    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-tmp\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319368    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-tmp-dir\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319390    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-etc-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319458    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/11b7246b-031f-46cd-ab96-814d2e5d4a72-audit-log\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319464    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9634e11-0c6c-4832-95a9-6665d679e1fd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319524    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a1f00d4-439a-46b1-98fe-b595af207890-tmp\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319553    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-tmp-dir\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319588    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-config\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319619    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b87353-bd2b-4a31-a5d3-9aa7582e0354-metrics-client-ca\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319673    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-var-lib-kubelet\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319690    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-client-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319703    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319732    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f26td\" (UniqueName: \"kubernetes.io/projected/51410712-fd08-4429-8d93-c0266660f335-kube-api-access-f26td\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319760    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-tls\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319808    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319839    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-systemd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319844    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319939    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-log-socket\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319950    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/54b87353-bd2b-4a31-a5d3-9aa7582e0354-secret-grpc-tls\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.319970    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320002    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320026    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320027    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320057    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbnh\" (UniqueName: \"kubernetes.io/projected/71f6bbd9-acd8-4534-8adb-ca7cd139d383-kube-api-access-jqbnh\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320094    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-env-overrides\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320246    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88501d15-b241-4af1-af7a-eabc4c417a08-konnectivity-ca\") pod \"konnectivity-agent-9d2nl\" (UID: \"88501d15-b241-4af1-af7a-eabc4c417a08\") " pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320301    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-tls\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320349    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/11b7246b-031f-46cd-ab96-814d2e5d4a72-secret-metrics-server-tls\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320440    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71f6bbd9-acd8-4534-8adb-ca7cd139d383-ovnkube-config\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320504    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b7246b-031f-46cd-ab96-814d2e5d4a72-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320660    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-tls\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320674    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54b87353-bd2b-4a31-a5d3-9aa7582e0354-metrics-client-ca\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320703    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320728    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320842    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.320893    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.321631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.321331    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zvn\" (UniqueName: \"kubernetes.io/projected/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-kube-api-access-89zvn\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:29:17.323434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.322204    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qrlf\" (UniqueName: \"kubernetes.io/projected/64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b-kube-api-access-9qrlf\") pod \"network-metrics-daemon-wm4x2\" (UID: \"64c3b8bb-f1d4-4a55-a1ee-cba3e77c6d3b\") " pod="openshift-multus/network-metrics-daemon-wm4x2"
Nov 03 10:29:17.323434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.322387    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.323434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.322399    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbvbl\" (UniqueName: \"kubernetes.io/projected/54b87353-bd2b-4a31-a5d3-9aa7582e0354-kube-api-access-wbvbl\") pod \"thanos-querier-76b7876cf5-7mrjf\" (UID: \"54b87353-bd2b-4a31-a5d3-9aa7582e0354\") " pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:17.323434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.322837    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6r65\" (UniqueName: \"kubernetes.io/projected/59e913f7-bc17-4001-bed8-348466e5966c-kube-api-access-g6r65\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.323434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.323088    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh24b\" (UniqueName: \"kubernetes.io/projected/cbcaff87-f298-42b7-8fce-0850c567d9c7-kube-api-access-fh24b\") pod \"network-check-target-cd6vm\" (UID: \"cbcaff87-f298-42b7-8fce-0850c567d9c7\") " pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:17.323434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.323159    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4wd\" (UniqueName: \"kubernetes.io/projected/11b7246b-031f-46cd-ab96-814d2e5d4a72-kube-api-access-fs4wd\") pod \"metrics-server-767d4fbf8c-86qwm\" (UID: \"11b7246b-031f-46cd-ab96-814d2e5d4a72\") " pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:17.323434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.323255    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcfs\" (UniqueName: \"kubernetes.io/projected/e0ecce6a-ad31-4a68-9298-5f5618ece509-kube-api-access-zhcfs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.323766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.323577    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4q2\" (UniqueName: \"kubernetes.io/projected/f9634e11-0c6c-4832-95a9-6665d679e1fd-kube-api-access-fn4q2\") pod \"monitoring-plugin-77f4f76594-ttks6\" (UID: \"f9634e11-0c6c-4832-95a9-6665d679e1fd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:17.324077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.324060    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqsg\" (UniqueName: \"kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.324715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.324695    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkggv\" (UniqueName: \"kubernetes.io/projected/2a1f00d4-439a-46b1-98fe-b595af207890-kube-api-access-tkggv\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.324829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.324695    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.324829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.324739    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6tf\" (UniqueName: \"kubernetes.io/projected/93cc76bb-77ed-493e-9207-c5bb8e30bc25-kube-api-access-4n6tf\") pod \"ingress-canary-kjzdq\" (UID: \"93cc76bb-77ed-493e-9207-c5bb8e30bc25\") " pod="openshift-ingress-canary/ingress-canary-kjzdq"
Nov 03 10:29:17.325053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.324985    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn96l\" (UniqueName: \"kubernetes.io/projected/17db0bdd-c610-416d-b20e-b0f51a839205-kube-api-access-bn96l\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.325178 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.325149    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69mv\" (UniqueName: \"kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.325282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.325262    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlmh\" (UniqueName: \"kubernetes.io/projected/20cf9d6c-4dc2-49e0-93c9-d6115eeeb905-kube-api-access-xhlmh\") pod \"dns-default-9zqgx\" (UID: \"20cf9d6c-4dc2-49e0-93c9-d6115eeeb905\") " pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:17.325350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.325310    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fvz\" (UniqueName: \"kubernetes.io/projected/fffd3590-b7f0-41c9-b24f-1575b39c8f25-kube-api-access-82fvz\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:29:17.325537 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.325514    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8hh\" (UniqueName: \"kubernetes.io/projected/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-kube-api-access-bg8hh\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.325618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.325605    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48f7q\" (UniqueName: \"kubernetes.io/projected/00141100-9f3c-4e38-b3d5-04a31b4c76e3-kube-api-access-48f7q\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.326769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.326747    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f26td\" (UniqueName: \"kubernetes.io/projected/51410712-fd08-4429-8d93-c0266660f335-kube-api-access-f26td\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:29:17.327180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.327165    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbnh\" (UniqueName: \"kubernetes.io/projected/71f6bbd9-acd8-4534-8adb-ca7cd139d383-kube-api-access-jqbnh\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.327408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.327392    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle\") pod \"console-59787cccbc-9wk85\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") " pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:17.330213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.330196    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:17.330796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.330776    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle\") pod \"console-576b7767c8-btzbj\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") " pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:17.421580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.421548    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.421740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.421590    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cnibin\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.421740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.421618    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-k8s-cni-cncf-io\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.421740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.421649    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-hostroot\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.421740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.421690    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-cnibin\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.421740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.421715    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-k8s-cni-cncf-io\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.421985    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-hostroot\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422041    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.422071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422068    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-bin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422090    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-host\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:29:17.422201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422115    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysconfig\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422137    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-lib-modules\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422158    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-host\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422160    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-bin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422182    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-ovn\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422203    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422205    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-sys\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422257    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422283    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-sys-fs\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422306    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-lib-modules\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422311    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-etc-kubernetes\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422347    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-netd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422353    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-etc-kubernetes\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422365    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-systemd\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422387    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-multus\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422406    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-kubelet-config\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422407    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-host\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422432    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-system-cni-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422437    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-kubelet-config\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422457    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e913f7-bc17-4001-bed8-348466e5966c-crio-socket\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422468    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7532f5ee-e2ed-40a2-9d9e-277a392e1c9f-host\") pod \"node-ca-67pqq\" (UID: \"7532f5ee-e2ed-40a2-9d9e-277a392e1c9f\") " pod="openshift-image-registry/node-ca-67pqq"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422482    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-dbus\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422509    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-slash\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422512    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysconfig\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422550    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-modprobe-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422553    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-slash\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422585    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-ovn\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422598    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-netd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422608    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-cnibin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422639    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422662    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-root\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422669    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-cni-multus\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.422685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422686    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-socket-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422730    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422756    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-modprobe-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422713    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-systemd-units\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422779    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-systemd-units\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422791    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-node-log\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422813    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430e1cec-2732-4742-8eed-d2ba0fc4caf3-dbus\") pod \"global-pull-secret-syncer-w4txd\" (UID: \"430e1cec-2732-4742-8eed-d2ba0fc4caf3\") " pod="kube-system/global-pull-secret-syncer-w4txd"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422820    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-wtmp\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422829    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-cnibin\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422845    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422862    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-sys\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422874    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-netns\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422876    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422640    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-systemd\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422937    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-root\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422941    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-kubernetes\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422940    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422960    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-node-exporter-wtmp\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422974    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422982    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-netns\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.422991    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-sys-fs\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423000    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-node-log\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423017    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423023    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423047    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-conf\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423071    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-kubelet\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423073    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423098    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-kubelet\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423123    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-netns\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423047    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e913f7-bc17-4001-bed8-348466e5966c-crio-socket\") pod \"insights-runtime-extractor-98krd\" (UID: \"59e913f7-bc17-4001-bed8-348466e5966c\") " pod="openshift-insights/insights-runtime-extractor-98krd"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423142    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-kubernetes\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423147    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-sys\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423172    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423178    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d16fcb39-bfc9-4d51-af89-14d83a3d07a1-sys\") pod \"node-exporter-78k5n\" (UID: \"d16fcb39-bfc9-4d51-af89-14d83a3d07a1\") " pod="openshift-monitoring/node-exporter-78k5n"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423169    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-d\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423077    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-system-cni-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423216    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-kubelet\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423219    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-run-netns\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423125    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-socket-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423225    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00141100-9f3c-4e38-b3d5-04a31b4c76e3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb997\" (UID: \"00141100-9f3c-4e38-b3d5-04a31b4c76e3\") " pod="openshift-multus/multus-additional-cni-plugins-nb997"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423247    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-var-lib-kubelet\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423248    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-etc-sysctl-conf\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.423249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423236    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-run\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423284    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-conf-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423304    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-socket-dir-parent\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423320    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-system-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423334    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-bin\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423342    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-conf-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423352    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51410712-fd08-4429-8d93-c0266660f335-hosts-file\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423371    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-var-lib-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423386    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-system-cni-dir\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423369    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-multus-socket-dir-parent\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423257    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-run\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423410    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-multus-certs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423386    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0ecce6a-ad31-4a68-9298-5f5618ece509-host-run-multus-certs\") pod \"multus-j7kpq\" (UID: \"e0ecce6a-ad31-4a68-9298-5f5618ece509\") " pod="openshift-multus/multus-j7kpq"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423437    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-registration-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423442    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-host-cni-bin\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423414    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51410712-fd08-4429-8d93-c0266660f335-hosts-file\") pod \"node-resolver-crz98\" (UID: \"51410712-fd08-4429-8d93-c0266660f335\") " pod="openshift-dns/node-resolver-crz98"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423461    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-device-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423478    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fffd3590-b7f0-41c9-b24f-1575b39c8f25-host-slash\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423497    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-etc-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423511    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-registration-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423518    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-var-lib-kubelet\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423533    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17db0bdd-c610-416d-b20e-b0f51a839205-device-dir\") pod \"aws-ebs-csi-driver-node-5mxw9\" (UID: \"17db0bdd-c610-416d-b20e-b0f51a839205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mxw9"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423556    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-etc-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423563    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a1f00d4-439a-46b1-98fe-b595af207890-var-lib-kubelet\") pod \"tuned-vgs7g\" (UID: \"2a1f00d4-439a-46b1-98fe-b595af207890\") " pod="openshift-cluster-node-tuning-operator/tuned-vgs7g"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423563    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fffd3590-b7f0-41c9-b24f-1575b39c8f25-host-slash\") pod \"iptables-alerter-9ft6m\" (UID: \"fffd3590-b7f0-41c9-b24f-1575b39c8f25\") " pod="openshift-network-operator/iptables-alerter-9ft6m"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423585    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-systemd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423606    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-log-socket\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423629    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-run-systemd\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423662    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-log-socket\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:17.424644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:17.423450    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f6bbd9-acd8-4534-8adb-ca7cd139d383-var-lib-openvswitch\") pod \"ovnkube-node-6t57h\" (UID: \"71f6bbd9-acd8-4534-8adb-ca7cd139d383\") " pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:18.178893 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:18.178857    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:18.180185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:18.180163    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9zqgx"
Nov 03 10:29:19.190128 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:19.190095    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:19.195669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:19.195648    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76b7876cf5-7mrjf"
Nov 03 10:29:19.699538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:19.699504    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:19.703020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:19.702995    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ttks6"
Nov 03 10:29:19.807031 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:19.806998    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:20.394249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:20.394213    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9d2nl"
Nov 03 10:29:22.734300 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:22.734272    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:22.890016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:22.889990    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:22.890192 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:22.890182    5393 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 10:29:24.177775 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:24.177739    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:25.878066 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:25.878024    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:27.117513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:27.117473    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6t57h"
Nov 03 10:29:30.629333 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:30.629296    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:30.717124 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:30.717091    5393 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9d2nl_88501d15-b241-4af1-af7a-eabc4c417a08/konnectivity-agent/0.log" limit=1024
Nov 03 10:29:34.182749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:34.182720    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:34.186383 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:34.186360    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:29:37.074306 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:37.074267    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:42.224829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.224779    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-576b7767c8-btzbj" podUID="87a6abe6-d585-4c92-9ee8-2707962af39b" containerName="console" containerID="cri-o://0f4d85593b948e960513f2853ced307f1669c7e68bf48ad652b9302839470fa9" gracePeriod=15
Nov 03 10:29:42.389867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.389830    5393 generic.go:356] "Generic (PLEG): container finished" podID="87a6abe6-d585-4c92-9ee8-2707962af39b" containerID="0f4d85593b948e960513f2853ced307f1669c7e68bf48ad652b9302839470fa9" exitCode=0
Nov 03 10:29:42.390041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.389894    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576b7767c8-btzbj" event={"ID":"87a6abe6-d585-4c92-9ee8-2707962af39b","Type":"ContainerDied","Data":"0f4d85593b948e960513f2853ced307f1669c7e68bf48ad652b9302839470fa9"}
Nov 03 10:29:42.489616 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.489557    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:42.542728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.542695    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config\") pod \"87a6abe6-d585-4c92-9ee8-2707962af39b\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") "
Nov 03 10:29:42.542728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.542728    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle\") pod \"87a6abe6-d585-4c92-9ee8-2707962af39b\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") "
Nov 03 10:29:42.542996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.542784    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config\") pod \"87a6abe6-d585-4c92-9ee8-2707962af39b\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") "
Nov 03 10:29:42.542996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.542879    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca\") pod \"87a6abe6-d585-4c92-9ee8-2707962af39b\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") "
Nov 03 10:29:42.542996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.542912    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert\") pod \"87a6abe6-d585-4c92-9ee8-2707962af39b\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") "
Nov 03 10:29:42.542996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.542952    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert\") pod \"87a6abe6-d585-4c92-9ee8-2707962af39b\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") "
Nov 03 10:29:42.542996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.542974    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg\" (UniqueName: \"kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg\") pod \"87a6abe6-d585-4c92-9ee8-2707962af39b\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") "
Nov 03 10:29:42.543250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.543013    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp\") pod \"87a6abe6-d585-4c92-9ee8-2707962af39b\" (UID: \"87a6abe6-d585-4c92-9ee8-2707962af39b\") "
Nov 03 10:29:42.543303 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.543279    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config" pod "87a6abe6-d585-4c92-9ee8-2707962af39b" (UID: "87a6abe6-d585-4c92-9ee8-2707962af39b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:42.543303 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.543292    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca" pod "87a6abe6-d585-4c92-9ee8-2707962af39b" (UID: "87a6abe6-d585-4c92-9ee8-2707962af39b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:42.543377 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.543315    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle" pod "87a6abe6-d585-4c92-9ee8-2707962af39b" (UID: "87a6abe6-d585-4c92-9ee8-2707962af39b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:42.543468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.543417    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp" pod "87a6abe6-d585-4c92-9ee8-2707962af39b" (UID: "87a6abe6-d585-4c92-9ee8-2707962af39b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:29:42.543468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.543428    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert" pod "87a6abe6-d585-4c92-9ee8-2707962af39b" (UID: "87a6abe6-d585-4c92-9ee8-2707962af39b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:42.545301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.545270    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config" pod "87a6abe6-d585-4c92-9ee8-2707962af39b" (UID: "87a6abe6-d585-4c92-9ee8-2707962af39b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:42.545431 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.545409    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert" pod "87a6abe6-d585-4c92-9ee8-2707962af39b" (UID: "87a6abe6-d585-4c92-9ee8-2707962af39b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:42.545493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.545443    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg" pod "87a6abe6-d585-4c92-9ee8-2707962af39b" (UID: "87a6abe6-d585-4c92-9ee8-2707962af39b"). InnerVolumeSpecName "kube-api-access-xqqsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:29:42.643795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.643746    5393 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-oauth-config\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:42.643795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.643789    5393 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:42.643795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.643800    5393 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-oauth-serving-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:42.643795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.643808    5393 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87a6abe6-d585-4c92-9ee8-2707962af39b-console-serving-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:42.644082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.643817    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqqsg\" (UniqueName: \"kubernetes.io/projected/87a6abe6-d585-4c92-9ee8-2707962af39b-kube-api-access-xqqsg\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:42.644082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.643829    5393 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87a6abe6-d585-4c92-9ee8-2707962af39b-tmp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:42.644082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.643837    5393 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-console-config\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:42.644082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:42.643846    5393 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87a6abe6-d585-4c92-9ee8-2707962af39b-trusted-ca-bundle\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:43.186764 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:43.186725    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:29:43.395716 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:43.395687    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576b7767c8-btzbj"
Nov 03 10:29:43.396154 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:43.395679    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576b7767c8-btzbj" event={"ID":"87a6abe6-d585-4c92-9ee8-2707962af39b","Type":"ContainerDied","Data":"b285052e9cfd14cc452965839833e52aa2a82953597d8052e0e923e91c8c5579"}
Nov 03 10:29:43.396154 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:43.395818    5393 scope.go:122] "RemoveContainer" containerID="0f4d85593b948e960513f2853ced307f1669c7e68bf48ad652b9302839470fa9"
Nov 03 10:29:43.416047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:43.416026    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-576b7767c8-btzbj"]
Nov 03 10:29:43.418147 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:43.418129    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-576b7767c8-btzbj"]
Nov 03 10:29:44.228327 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:44.228298    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a6abe6-d585-4c92-9ee8-2707962af39b" path="/var/lib/kubelet/pods/87a6abe6-d585-4c92-9ee8-2707962af39b/volumes"
Nov 03 10:29:45.440476 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:45.440436    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:45.440979 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:45.440845    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="alertmanager" containerID="cri-o://593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004" gracePeriod=120
Nov 03 10:29:45.441060 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:45.440940    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy-metric" containerID="cri-o://50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd" gracePeriod=120
Nov 03 10:29:45.441060 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:45.440964    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="prom-label-proxy" containerID="cri-o://5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a" gracePeriod=120
Nov 03 10:29:45.441060 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:45.440997    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy" containerID="cri-o://0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45" gracePeriod=120
Nov 03 10:29:45.441184 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:45.440973    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy-web" containerID="cri-o://9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee" gracePeriod=120
Nov 03 10:29:45.441303 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:45.441245    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="config-reloader" containerID="cri-o://7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754" gracePeriod=120
Nov 03 10:29:46.411555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411521    5393 generic.go:356] "Generic (PLEG): container finished" podID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerID="5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a" exitCode=0
Nov 03 10:29:46.411555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411545    5393 generic.go:356] "Generic (PLEG): container finished" podID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerID="50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd" exitCode=0
Nov 03 10:29:46.411555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411558    5393 generic.go:356] "Generic (PLEG): container finished" podID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerID="0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45" exitCode=0
Nov 03 10:29:46.411796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411568    5393 generic.go:356] "Generic (PLEG): container finished" podID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerID="7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754" exitCode=0
Nov 03 10:29:46.411796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411576    5393 generic.go:356] "Generic (PLEG): container finished" podID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerID="593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004" exitCode=0
Nov 03 10:29:46.411796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411592    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerDied","Data":"5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a"}
Nov 03 10:29:46.411796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411657    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerDied","Data":"50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd"}
Nov 03 10:29:46.411796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411691    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerDied","Data":"0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45"}
Nov 03 10:29:46.411796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411715    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerDied","Data":"7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754"}
Nov 03 10:29:46.411796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:46.411738    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerDied","Data":"593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004"}
Nov 03 10:29:47.205816 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.205786    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.276323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276295    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276340    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276381    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276417    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276443    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276473    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276504    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276565    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276597    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276628    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276679    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276707    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.276813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.276737    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\" (UID: \"a5e57152-22c4-4ecb-bfb1-ff1a026307f3\") "
Nov 03 10:29:47.277288 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.277142    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:47.277863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.277569    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:29:47.278713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.278682    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:29:47.280201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.280132    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:47.280904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.280875    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:47.280904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.280892    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:29:47.281078 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.280987    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:47.281344 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.281321    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:29:47.281618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.281590    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "kube-api-access-zqhh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:29:47.282250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.282215    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:47.282515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.282496    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:47.285138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.284893    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:47.291792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.291768    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config" pod "a5e57152-22c4-4ecb-bfb1-ff1a026307f3" (UID: "a5e57152-22c4-4ecb-bfb1-ff1a026307f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:29:47.377510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377478    5393 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-tls-assets\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377506    5393 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-cluster-tls-config\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377517    5393 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-main-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377526    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-metrics-client-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377535    5393 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-out\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377545    5393 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377574    5393 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-main-db\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377583    5393 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377593    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqhh9\" (UniqueName: \"kubernetes.io/projected/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-kube-api-access-zqhh9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377601    5393 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377609    5393 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377618    5393 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.377731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.377627    5393 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5e57152-22c4-4ecb-bfb1-ff1a026307f3-web-config\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:29:47.418496 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.418467    5393 generic.go:356] "Generic (PLEG): container finished" podID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerID="9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee" exitCode=0
Nov 03 10:29:47.418654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.418516    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerDied","Data":"9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee"}
Nov 03 10:29:47.418654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.418546    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.418654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.418572    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5e57152-22c4-4ecb-bfb1-ff1a026307f3","Type":"ContainerDied","Data":"3015f5c63a4e021ad2a0c057693a9cb293a9e8524a0a8561ef113da0017915ed"}
Nov 03 10:29:47.418654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.418608    5393 scope.go:122] "RemoveContainer" containerID="5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a"
Nov 03 10:29:47.430957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.430938    5393 scope.go:122] "RemoveContainer" containerID="50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd"
Nov 03 10:29:47.437741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.437719    5393 scope.go:122] "RemoveContainer" containerID="0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45"
Nov 03 10:29:47.442655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.442622    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:47.445137 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.445054    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:47.447950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.447909    5393 scope.go:122] "RemoveContainer" containerID="9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee"
Nov 03 10:29:47.454671 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.454653    5393 scope.go:122] "RemoveContainer" containerID="7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754"
Nov 03 10:29:47.461490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.461474    5393 scope.go:122] "RemoveContainer" containerID="593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004"
Nov 03 10:29:47.468378 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468357    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:47.468655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468614    5393 cpu_manager.go:414] "containerMap: removing container" podUID="87a6abe6-d585-4c92-9ee8-2707962af39b" containerName="console"
Nov 03 10:29:47.468696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468657    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a6abe6-d585-4c92-9ee8-2707962af39b" containerName="console"
Nov 03 10:29:47.468696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468671    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="alertmanager"
Nov 03 10:29:47.468696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468677    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="alertmanager"
Nov 03 10:29:47.468696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468688    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="prom-label-proxy"
Nov 03 10:29:47.468696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468692    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="prom-label-proxy"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468722    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy-metric"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468727    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy-metric"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468736    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468741    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468749    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="init-config-reloader"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468753    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="init-config-reloader"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468760    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy-web"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468764    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy-web"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468769    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="config-reloader"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468773    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="config-reloader"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468812    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="prom-label-proxy"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468818    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy-metric"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468824    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468829    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="init-config-reloader"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468835    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="kube-rbac-proxy-web"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468840    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="config-reloader"
Nov 03 10:29:47.468842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468847    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a6abe6-d585-4c92-9ee8-2707962af39b" containerName="console"
Nov 03 10:29:47.469363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.468853    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" containerName="alertmanager"
Nov 03 10:29:47.469363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.469088    5393 scope.go:122] "RemoveContainer" containerID="1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c"
Nov 03 10:29:47.476430 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.476411    5393 scope.go:122] "RemoveContainer" containerID="5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a"
Nov 03 10:29:47.476685 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:47.476664    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a\": container with ID starting with 5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a not found: ID does not exist" containerID="5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a"
Nov 03 10:29:47.476749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.476695    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a"} err="failed to get container status \"5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a\": rpc error: code = NotFound desc = could not find container \"5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a\": container with ID starting with 5cc504e45f1d9feba377993dabdb476334dd7f58d920d76dd489dfcd7b314d5a not found: ID does not exist"
Nov 03 10:29:47.476749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.476737    5393 scope.go:122] "RemoveContainer" containerID="50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd"
Nov 03 10:29:47.476978 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:47.476963    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd\": container with ID starting with 50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd not found: ID does not exist" containerID="50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd"
Nov 03 10:29:47.477032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.476982    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd"} err="failed to get container status \"50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd\": rpc error: code = NotFound desc = could not find container \"50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd\": container with ID starting with 50fb64d11151864d827ca12ffdb06b4cfe318def117948fea3ad7c930850b3fd not found: ID does not exist"
Nov 03 10:29:47.477032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.476993    5393 scope.go:122] "RemoveContainer" containerID="0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45"
Nov 03 10:29:47.477214 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:47.477194    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45\": container with ID starting with 0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45 not found: ID does not exist" containerID="0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45"
Nov 03 10:29:47.477313 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.477215    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45"} err="failed to get container status \"0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45\": rpc error: code = NotFound desc = could not find container \"0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45\": container with ID starting with 0dec11af7baf36969d2f8fa7beed874a3c40fc7c89cdfe8a713795ed8a19df45 not found: ID does not exist"
Nov 03 10:29:47.477313 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.477231    5393 scope.go:122] "RemoveContainer" containerID="9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee"
Nov 03 10:29:47.477440 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:47.477425    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee\": container with ID starting with 9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee not found: ID does not exist" containerID="9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee"
Nov 03 10:29:47.477504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.477448    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee"} err="failed to get container status \"9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee\": rpc error: code = NotFound desc = could not find container \"9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee\": container with ID starting with 9f4617f23fe6600502b80a8801ff938323db86995bb85b0490bd909031d1a6ee not found: ID does not exist"
Nov 03 10:29:47.477504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.477465    5393 scope.go:122] "RemoveContainer" containerID="7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754"
Nov 03 10:29:47.477666 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:47.477648    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754\": container with ID starting with 7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754 not found: ID does not exist" containerID="7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754"
Nov 03 10:29:47.477699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.477672    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754"} err="failed to get container status \"7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754\": rpc error: code = NotFound desc = could not find container \"7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754\": container with ID starting with 7baabe1498d47127b88efdd3b2598e1513da005d938945248895d6adfe193754 not found: ID does not exist"
Nov 03 10:29:47.477699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.477687    5393 scope.go:122] "RemoveContainer" containerID="593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004"
Nov 03 10:29:47.477981 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:47.477964    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004\": container with ID starting with 593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004 not found: ID does not exist" containerID="593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004"
Nov 03 10:29:47.478023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.477987    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004"} err="failed to get container status \"593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004\": rpc error: code = NotFound desc = could not find container \"593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004\": container with ID starting with 593bf64af50cf9a22e3cdb546902a546e52e6ab267075a21e84b03f96bd14004 not found: ID does not exist"
Nov 03 10:29:47.478023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.478001    5393 scope.go:122] "RemoveContainer" containerID="1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c"
Nov 03 10:29:47.478209 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:29:47.478192    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c\": container with ID starting with 1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c not found: ID does not exist" containerID="1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c"
Nov 03 10:29:47.478254 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.478215    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c"} err="failed to get container status \"1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c\": rpc error: code = NotFound desc = could not find container \"1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c\": container with ID starting with 1dfbe787127f64ef116e6a2e375d52465683a20900d648041358466aedfd155c not found: ID does not exist"
Nov 03 10:29:47.500054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.500025    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:47.500185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.500150    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.507419 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.507388    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 10:29:47.507419 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.507411    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 10:29:47.507551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.507490    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 10:29:47.507551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.507490    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 10:29:47.507728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.507712    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 10:29:47.507728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.507719    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 10:29:47.507827 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.507743    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 10:29:47.508869 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.508854    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 10:29:47.508954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.508889    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fnqcd\""
Nov 03 10:29:47.512101 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.512082    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 10:29:47.579003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.578968    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579004    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-web-config\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579031    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnd9r\" (UniqueName: \"kubernetes.io/projected/c37a9918-badd-4dc3-b824-58b547004e22-kube-api-access-jnd9r\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579144    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c37a9918-badd-4dc3-b824-58b547004e22-config-out\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579195    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c37a9918-badd-4dc3-b824-58b547004e22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579303 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579223    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-config-volume\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579303 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579244    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579303 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579259    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579383 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579335    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c37a9918-badd-4dc3-b824-58b547004e22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579383 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579362    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c37a9918-badd-4dc3-b824-58b547004e22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579383 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579377    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c37a9918-badd-4dc3-b824-58b547004e22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579392    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.579499 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.579471    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680103    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c37a9918-badd-4dc3-b824-58b547004e22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680138    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c37a9918-badd-4dc3-b824-58b547004e22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680157    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680179    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680198    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680313    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-web-config\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680359    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnd9r\" (UniqueName: \"kubernetes.io/projected/c37a9918-badd-4dc3-b824-58b547004e22-kube-api-access-jnd9r\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680400    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c37a9918-badd-4dc3-b824-58b547004e22-config-out\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680424    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c37a9918-badd-4dc3-b824-58b547004e22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680457    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-config-volume\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680491    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680836 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680513    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.680836 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680561    5393 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 10:29:47.680836 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680570    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c37a9918-badd-4dc3-b824-58b547004e22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.681026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680851    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c37a9918-badd-4dc3-b824-58b547004e22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.681026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.680892    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c37a9918-badd-4dc3-b824-58b547004e22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.681451 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.681379    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c37a9918-badd-4dc3-b824-58b547004e22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.683384 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.683362    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c37a9918-badd-4dc3-b824-58b547004e22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.683654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.683584    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c37a9918-badd-4dc3-b824-58b547004e22-config-out\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.684180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.683902    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.684180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.684160    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-config-volume\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.684297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.684160    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.684297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.684260    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-web-config\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.684372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.684356    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.684421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.684406    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.685733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.685716    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c37a9918-badd-4dc3-b824-58b547004e22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.689621 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.689604    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnd9r\" (UniqueName: \"kubernetes.io/projected/c37a9918-badd-4dc3-b824-58b547004e22-kube-api-access-jnd9r\") pod \"alertmanager-main-0\" (UID: \"c37a9918-badd-4dc3-b824-58b547004e22\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:47.869677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:47.869639    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 10:29:48.164315 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:48.164292    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 10:29:48.165566 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:48.165535    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37a9918_badd_4dc3_b824_58b547004e22.slice/crio-fd33755e7df34673877d29aac5a03abac48292d58a901eab700b389ba0593f5a WatchSource:0}: Error finding container fd33755e7df34673877d29aac5a03abac48292d58a901eab700b389ba0593f5a: Status 404 returned error can't find the container with id fd33755e7df34673877d29aac5a03abac48292d58a901eab700b389ba0593f5a
Nov 03 10:29:48.227663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:48.227634    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e57152-22c4-4ecb-bfb1-ff1a026307f3" path="/var/lib/kubelet/pods/a5e57152-22c4-4ecb-bfb1-ff1a026307f3/volumes"
Nov 03 10:29:48.424342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:48.424306    5393 generic.go:356] "Generic (PLEG): container finished" podID="c37a9918-badd-4dc3-b824-58b547004e22" containerID="94145cce8c17561ba5ec0ce81612b08cb36f5364d65dfd1673db24fb1c6354a6" exitCode=0
Nov 03 10:29:48.424566 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:48.424367    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c37a9918-badd-4dc3-b824-58b547004e22","Type":"ContainerDied","Data":"94145cce8c17561ba5ec0ce81612b08cb36f5364d65dfd1673db24fb1c6354a6"}
Nov 03 10:29:48.424566 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:48.424423    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c37a9918-badd-4dc3-b824-58b547004e22","Type":"ContainerStarted","Data":"fd33755e7df34673877d29aac5a03abac48292d58a901eab700b389ba0593f5a"}
Nov 03 10:29:49.432207 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.432172    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c37a9918-badd-4dc3-b824-58b547004e22","Type":"ContainerStarted","Data":"02c75922f3dacfdd9211366c8e2338c4d59c367739c430f17333ee10942327ed"}
Nov 03 10:29:49.432600 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.432260    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c37a9918-badd-4dc3-b824-58b547004e22","Type":"ContainerStarted","Data":"ace22152e868d40c54c26f29a6ef9259b580211a11c13a83563d04aebc6eb418"}
Nov 03 10:29:49.432600 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.432292    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c37a9918-badd-4dc3-b824-58b547004e22","Type":"ContainerStarted","Data":"abcfef30530b866b296c9565bf208972fdaccdfd9cfff4468af0165dff80fd4b"}
Nov 03 10:29:49.432600 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.432318    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c37a9918-badd-4dc3-b824-58b547004e22","Type":"ContainerStarted","Data":"e33a7dcaf91568d5e4a56b9ea44771488017fed32fafb932f201f60457f48208"}
Nov 03 10:29:49.432600 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.432341    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c37a9918-badd-4dc3-b824-58b547004e22","Type":"ContainerStarted","Data":"028ad6a0dfe19c4a28501f6da61f02ced5568f035f03f93b9bb30302b40f0749"}
Nov 03 10:29:49.432600 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.432361    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c37a9918-badd-4dc3-b824-58b547004e22","Type":"ContainerStarted","Data":"e7563629364863406d546776077aa3c42a12bc252f74e06d64caf7ce904f0c5d"}
Nov 03 10:29:49.458533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.458487    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.444380606 podStartE2EDuration="2.458472678s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.092072ms" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:49.456543456 +0000 UTC m=+33.361397915" watchObservedRunningTime="2026-11-03 10:29:49.458472678 +0000 UTC m=+33.363327118"
Nov 03 10:29:49.461388 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.461365    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"]
Nov 03 10:29:49.479759 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.479729    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.480222 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.480194    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"]
Nov 03 10:29:49.487953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.483823    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 10:29:49.487953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.487150    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 10:29:49.494380 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.494354    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-secret-telemeter-client\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.494493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.494394    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxllp\" (UniqueName: \"kubernetes.io/projected/e88024db-34c5-4f34-b838-79feb2aa83fa-kube-api-access-wxllp\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.494493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.494449    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.494733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.494699    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.494840 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.494762    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-federate-client-tls\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.494840 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.494815    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-serving-certs-ca-bundle\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.494989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.494910    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-metrics-client-ca\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.495041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.494987    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-telemeter-client-tls\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.561136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.561101    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 10:29:49.561136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.561101    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 10:29:49.561136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.561105    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 10:29:49.561346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.561147    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qhc45\""
Nov 03 10:29:49.561346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.561111    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 10:29:49.595518 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.595486    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-federate-client-tls\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.595518 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.595520    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-serving-certs-ca-bundle\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.595746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.595540    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-metrics-client-ca\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.595746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.595558    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-telemeter-client-tls\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.595746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.595582    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-secret-telemeter-client\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.595746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.595600    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxllp\" (UniqueName: \"kubernetes.io/projected/e88024db-34c5-4f34-b838-79feb2aa83fa-kube-api-access-wxllp\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.595746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.595638    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.595746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.595684    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.596350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.596320    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-metrics-client-ca\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.596443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.596424    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-serving-certs-ca-bundle\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.596878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.596854    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e88024db-34c5-4f34-b838-79feb2aa83fa-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.598388 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.598364    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-telemeter-client-tls\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.598469 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.598424    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.598626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.598605    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-federate-client-tls\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.598661 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.598619    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e88024db-34c5-4f34-b838-79feb2aa83fa-secret-telemeter-client\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.603170 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.603152    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxllp\" (UniqueName: \"kubernetes.io/projected/e88024db-34c5-4f34-b838-79feb2aa83fa-kube-api-access-wxllp\") pod \"telemeter-client-5d7db6cd45-cj2lf\" (UID: \"e88024db-34c5-4f34-b838-79feb2aa83fa\") " pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:49.831974 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:49.831894    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"
Nov 03 10:29:50.140100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:50.140073    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf"]
Nov 03 10:29:50.141518 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:50.141491    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88024db_34c5_4f34_b838_79feb2aa83fa.slice/crio-63f5d00af2b736634631a92f2cb9344e1d4f6a57409f0da2be717c81fdd53a8c WatchSource:0}: Error finding container 63f5d00af2b736634631a92f2cb9344e1d4f6a57409f0da2be717c81fdd53a8c: Status 404 returned error can't find the container with id 63f5d00af2b736634631a92f2cb9344e1d4f6a57409f0da2be717c81fdd53a8c
Nov 03 10:29:50.143557 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:50.143537    5393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:29:50.437886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:50.437790    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf" event={"ID":"e88024db-34c5-4f34-b838-79feb2aa83fa","Type":"ContainerStarted","Data":"63f5d00af2b736634631a92f2cb9344e1d4f6a57409f0da2be717c81fdd53a8c"}
Nov 03 10:29:52.448085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:52.448051    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf" event={"ID":"e88024db-34c5-4f34-b838-79feb2aa83fa","Type":"ContainerStarted","Data":"21deea00464b47d6434ab6acb2ea6e91d6ea63540f86b290ef6a922353683c38"}
Nov 03 10:29:52.737645 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:52.737618    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cd6vm"
Nov 03 10:29:53.453648 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.453610    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf" event={"ID":"e88024db-34c5-4f34-b838-79feb2aa83fa","Type":"ContainerStarted","Data":"35a01a2d51580c04a90b15c7694fb081ecb32991c56baaa76b35b402c6fe0b36"}
Nov 03 10:29:53.454084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.453675    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf" event={"ID":"e88024db-34c5-4f34-b838-79feb2aa83fa","Type":"ContainerStarted","Data":"36e83b4af55af8ab22b2459465356ed9f27bf17f52df988895c673312c62e64f"}
Nov 03 10:29:53.476514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.476461    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d7db6cd45-cj2lf" podStartSLOduration=2.280873335 podStartE2EDuration="4.476445813s" totalImagesPullingTime="2.195572478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:53.475875203 +0000 UTC m=+37.380729643" watchObservedRunningTime="2026-11-03 10:29:53.476445813 +0000 UTC m=+37.381300254"
Nov 03 10:29:53.554973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.554943    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d466b654-9zg4p"]
Nov 03 10:29:53.557210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.557194    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.567634 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.567600    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d466b654-9zg4p"]
Nov 03 10:29:53.629766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.629707    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-oauth-config\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.629766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.629772    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-config\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.630068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.629869    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-serving-cert\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.630068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.629903    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbd0206-072c-4a1f-8cec-1de48f6819c1-tmp\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.630068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.629969    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-oauth-serving-cert\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.630068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.629994    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzp8\" (UniqueName: \"kubernetes.io/projected/0cbd0206-072c-4a1f-8cec-1de48f6819c1-kube-api-access-vlzp8\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.630188 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.630099    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-trusted-ca-bundle\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.630188 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.630137    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-service-ca\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.731447 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.731369    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-trusted-ca-bundle\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.731447 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.731410    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-service-ca\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.731592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.731452    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-oauth-config\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.731592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.731490    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-config\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.731592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.731523    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-serving-cert\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.731592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.731563    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbd0206-072c-4a1f-8cec-1de48f6819c1-tmp\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.731854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.731595    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-oauth-serving-cert\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.731854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.731624    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzp8\" (UniqueName: \"kubernetes.io/projected/0cbd0206-072c-4a1f-8cec-1de48f6819c1-kube-api-access-vlzp8\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.732067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.732040    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbd0206-072c-4a1f-8cec-1de48f6819c1-tmp\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.732354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.732266    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-service-ca\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.732354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.732289    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-config\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.732618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.732444    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-oauth-serving-cert\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.732618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.732599    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-trusted-ca-bundle\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.734352 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.734330    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-oauth-config\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.734436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.734330    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-serving-cert\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.739669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.739650    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzp8\" (UniqueName: \"kubernetes.io/projected/0cbd0206-072c-4a1f-8cec-1de48f6819c1-kube-api-access-vlzp8\") pod \"console-67d466b654-9zg4p\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") " pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:53.868475 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:53.868437    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:29:54.136885 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:54.136737    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d466b654-9zg4p"]
Nov 03 10:29:54.139430 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:29:54.139403    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbd0206_072c_4a1f_8cec_1de48f6819c1.slice/crio-20730269d32a7da99e953384b5f7921536c412bf68ebe8be3b1167e0406af4f7 WatchSource:0}: Error finding container 20730269d32a7da99e953384b5f7921536c412bf68ebe8be3b1167e0406af4f7: Status 404 returned error can't find the container with id 20730269d32a7da99e953384b5f7921536c412bf68ebe8be3b1167e0406af4f7
Nov 03 10:29:54.462797 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:54.460225    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d466b654-9zg4p" event={"ID":"0cbd0206-072c-4a1f-8cec-1de48f6819c1","Type":"ContainerStarted","Data":"b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c"}
Nov 03 10:29:54.462797 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:54.460319    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d466b654-9zg4p" event={"ID":"0cbd0206-072c-4a1f-8cec-1de48f6819c1","Type":"ContainerStarted","Data":"20730269d32a7da99e953384b5f7921536c412bf68ebe8be3b1167e0406af4f7"}
Nov 03 10:29:54.478991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:54.478938    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67d466b654-9zg4p" podStartSLOduration=1.478901067 podStartE2EDuration="1.478901067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:29:54.47684951 +0000 UTC m=+38.381703953" watchObservedRunningTime="2026-11-03 10:29:54.478901067 +0000 UTC m=+38.383755501"
Nov 03 10:29:57.079357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:57.079282    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:29:57.082950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:29:57.082931    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-767d4fbf8c-86qwm"
Nov 03 10:30:03.868670 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:03.868632    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:30:03.869209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:03.868819    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:30:03.873778 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:03.873746    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:30:04.499692 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:04.499669    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:30:04.541436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:04.541399    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59787cccbc-9wk85"]
Nov 03 10:30:13.187756 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:30:13.187717    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:30:29.561195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.561151    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59787cccbc-9wk85" podUID="856fad64-8c12-4623-b444-5d99425a7505" containerName="console" containerID="cri-o://7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750" gracePeriod=15
Nov 03 10:30:29.808509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.808485    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:30:29.919149 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919120    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca\") pod \"856fad64-8c12-4623-b444-5d99425a7505\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") "
Nov 03 10:30:29.919360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919169    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle\") pod \"856fad64-8c12-4623-b444-5d99425a7505\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") "
Nov 03 10:30:29.919360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919206    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert\") pod \"856fad64-8c12-4623-b444-5d99425a7505\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") "
Nov 03 10:30:29.919360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919236    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp\" (UniqueName: \"kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp\") pod \"856fad64-8c12-4623-b444-5d99425a7505\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") "
Nov 03 10:30:29.919360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919258    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv\" (UniqueName: \"kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv\") pod \"856fad64-8c12-4623-b444-5d99425a7505\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") "
Nov 03 10:30:29.919360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919285    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config\") pod \"856fad64-8c12-4623-b444-5d99425a7505\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") "
Nov 03 10:30:29.919360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919304    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config\") pod \"856fad64-8c12-4623-b444-5d99425a7505\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") "
Nov 03 10:30:29.919360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919334    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert\") pod \"856fad64-8c12-4623-b444-5d99425a7505\" (UID: \"856fad64-8c12-4623-b444-5d99425a7505\") "
Nov 03 10:30:29.919705 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919561    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp" pod "856fad64-8c12-4623-b444-5d99425a7505" (UID: "856fad64-8c12-4623-b444-5d99425a7505"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:30:29.919705 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919615    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca" pod "856fad64-8c12-4623-b444-5d99425a7505" (UID: "856fad64-8c12-4623-b444-5d99425a7505"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:30:29.919801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919696    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert" pod "856fad64-8c12-4623-b444-5d99425a7505" (UID: "856fad64-8c12-4623-b444-5d99425a7505"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:30:29.919976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.919847    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config" pod "856fad64-8c12-4623-b444-5d99425a7505" (UID: "856fad64-8c12-4623-b444-5d99425a7505"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:30:29.920064 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.920007    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle" pod "856fad64-8c12-4623-b444-5d99425a7505" (UID: "856fad64-8c12-4623-b444-5d99425a7505"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:30:29.921678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.921654    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert" pod "856fad64-8c12-4623-b444-5d99425a7505" (UID: "856fad64-8c12-4623-b444-5d99425a7505"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:30:29.921678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.921669    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv" pod "856fad64-8c12-4623-b444-5d99425a7505" (UID: "856fad64-8c12-4623-b444-5d99425a7505"). InnerVolumeSpecName "kube-api-access-m69mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:30:29.921814 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:29.921717    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config" pod "856fad64-8c12-4623-b444-5d99425a7505" (UID: "856fad64-8c12-4623-b444-5d99425a7505"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:30:30.020244 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.020203    5393 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-oauth-serving-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:30:30.020244 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.020239    5393 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856fad64-8c12-4623-b444-5d99425a7505-tmp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:30:30.020465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.020252    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m69mv\" (UniqueName: \"kubernetes.io/projected/856fad64-8c12-4623-b444-5d99425a7505-kube-api-access-m69mv\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:30:30.020465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.020300    5393 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-oauth-config\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:30:30.020465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.020316    5393 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-console-config\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:30:30.020465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.020329    5393 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/856fad64-8c12-4623-b444-5d99425a7505-console-serving-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:30:30.020465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.020341    5393 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:30:30.020465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.020352    5393 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/856fad64-8c12-4623-b444-5d99425a7505-trusted-ca-bundle\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:30:30.589149 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.589066    5393 generic.go:356] "Generic (PLEG): container finished" podID="856fad64-8c12-4623-b444-5d99425a7505" containerID="7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750" exitCode=0
Nov 03 10:30:30.589149 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.589112    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59787cccbc-9wk85"
Nov 03 10:30:30.589149 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.589122    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59787cccbc-9wk85" event={"ID":"856fad64-8c12-4623-b444-5d99425a7505","Type":"ContainerDied","Data":"7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750"}
Nov 03 10:30:30.589666 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.589161    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59787cccbc-9wk85" event={"ID":"856fad64-8c12-4623-b444-5d99425a7505","Type":"ContainerDied","Data":"f2c19c519c02b28a65f483c6dccbbe9b22e3d77b809e98972034f7198ea71974"}
Nov 03 10:30:30.589666 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.589182    5393 scope.go:122] "RemoveContainer" containerID="7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750"
Nov 03 10:30:30.597261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.597244    5393 scope.go:122] "RemoveContainer" containerID="7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750"
Nov 03 10:30:30.597524 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:30:30.597504    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750\": container with ID starting with 7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750 not found: ID does not exist" containerID="7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750"
Nov 03 10:30:30.597566 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.597532    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750"} err="failed to get container status \"7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750\": rpc error: code = NotFound desc = could not find container \"7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750\": container with ID starting with 7e6f810a920b2c257075504367580a847ad3edcb2a837bc100b3d166580e5750 not found: ID does not exist"
Nov 03 10:30:30.605280 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.605259    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59787cccbc-9wk85"]
Nov 03 10:30:30.608570 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:30.608552    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59787cccbc-9wk85"]
Nov 03 10:30:32.227444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:32.227401    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856fad64-8c12-4623-b444-5d99425a7505" path="/var/lib/kubelet/pods/856fad64-8c12-4623-b444-5d99425a7505/volumes"
Nov 03 10:30:43.187102 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:30:43.187059    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:30:44.354622 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.354592    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dd74748b9-c9n62"]
Nov 03 10:30:44.355032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.354861    5393 cpu_manager.go:414] "containerMap: removing container" podUID="856fad64-8c12-4623-b444-5d99425a7505" containerName="console"
Nov 03 10:30:44.355032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.354871    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856fad64-8c12-4623-b444-5d99425a7505" containerName="console"
Nov 03 10:30:44.355032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.354967    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="856fad64-8c12-4623-b444-5d99425a7505" containerName="console"
Nov 03 10:30:44.359006 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.358987    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.367270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.367234    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd74748b9-c9n62"]
Nov 03 10:30:44.436628 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.436595    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32e9ac83-284d-4d25-bb73-771438efc6c5-console-oauth-config\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.436628 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.436632    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-service-ca\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.436851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.436650    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-trusted-ca-bundle\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.436851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.436767    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e9ac83-284d-4d25-bb73-771438efc6c5-tmp\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.436851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.436804    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-oauth-serving-cert\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.436851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.436824    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32e9ac83-284d-4d25-bb73-771438efc6c5-console-serving-cert\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.436851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.436839    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-console-config\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.437109 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.437013    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjdd\" (UniqueName: \"kubernetes.io/projected/32e9ac83-284d-4d25-bb73-771438efc6c5-kube-api-access-6bjdd\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.538343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.538312    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-service-ca\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.538343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.538347    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-trusted-ca-bundle\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.538597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.538364    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e9ac83-284d-4d25-bb73-771438efc6c5-tmp\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.538597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.538488    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-oauth-serving-cert\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.538597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.538532    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32e9ac83-284d-4d25-bb73-771438efc6c5-console-serving-cert\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.538597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.538553    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-console-config\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.538796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.538642    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjdd\" (UniqueName: \"kubernetes.io/projected/32e9ac83-284d-4d25-bb73-771438efc6c5-kube-api-access-6bjdd\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.538796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.538689    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32e9ac83-284d-4d25-bb73-771438efc6c5-console-oauth-config\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.539268 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.539234    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-service-ca\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.539564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.539267    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-oauth-serving-cert\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.539564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.539346    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-trusted-ca-bundle\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.539564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.539347    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32e9ac83-284d-4d25-bb73-771438efc6c5-console-config\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.539564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.539383    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e9ac83-284d-4d25-bb73-771438efc6c5-tmp\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.541096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.541075    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32e9ac83-284d-4d25-bb73-771438efc6c5-console-serving-cert\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.541276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.541257    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32e9ac83-284d-4d25-bb73-771438efc6c5-console-oauth-config\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.546666 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.546645    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjdd\" (UniqueName: \"kubernetes.io/projected/32e9ac83-284d-4d25-bb73-771438efc6c5-kube-api-access-6bjdd\") pod \"console-7dd74748b9-c9n62\" (UID: \"32e9ac83-284d-4d25-bb73-771438efc6c5\") " pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.671194 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.671107    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:44.941214 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:44.941149    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd74748b9-c9n62"]
Nov 03 10:30:44.944476 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:30:44.944448    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e9ac83_284d_4d25_bb73_771438efc6c5.slice/crio-30399d093ecfb3daf454484ca4526a262c86e1711e322732298971093cd79ee3 WatchSource:0}: Error finding container 30399d093ecfb3daf454484ca4526a262c86e1711e322732298971093cd79ee3: Status 404 returned error can't find the container with id 30399d093ecfb3daf454484ca4526a262c86e1711e322732298971093cd79ee3
Nov 03 10:30:45.643024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:45.642989    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd74748b9-c9n62" event={"ID":"32e9ac83-284d-4d25-bb73-771438efc6c5","Type":"ContainerStarted","Data":"2fa06ccb303bfe65e89fee963016f863078b05a23c7a94b6b59840634dbbb709"}
Nov 03 10:30:45.643409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:45.643038    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd74748b9-c9n62" event={"ID":"32e9ac83-284d-4d25-bb73-771438efc6c5","Type":"ContainerStarted","Data":"30399d093ecfb3daf454484ca4526a262c86e1711e322732298971093cd79ee3"}
Nov 03 10:30:45.662766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:45.662719    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7dd74748b9-c9n62" podStartSLOduration=1.662705495 podStartE2EDuration="1.662705495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:30:45.661389053 +0000 UTC m=+89.566243496" watchObservedRunningTime="2026-11-03 10:30:45.662705495 +0000 UTC m=+89.567559930"
Nov 03 10:30:54.671942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:54.671876    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:54.672421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:54.671973    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:54.679270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:54.678150    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:54.683226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:54.683204    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dd74748b9-c9n62"
Nov 03 10:30:54.738509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:30:54.738479    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d466b654-9zg4p"]
Nov 03 10:31:13.186833 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:31:13.186798    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:31:19.758568 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:19.758504    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67d466b654-9zg4p" podUID="0cbd0206-072c-4a1f-8cec-1de48f6819c1" containerName="console" containerID="cri-o://b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c" gracePeriod=15
Nov 03 10:31:19.998350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:19.998326    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:31:20.123825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.123772    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbd0206-072c-4a1f-8cec-1de48f6819c1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbd0206-072c-4a1f-8cec-1de48f6819c1-tmp\") pod \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") "
Nov 03 10:31:20.124047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.123868    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-oauth-config\") pod \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") "
Nov 03 10:31:20.124047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.123903    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbd0206-072c-4a1f-8cec-1de48f6819c1-kube-api-access-vlzp8\" (UniqueName: \"kubernetes.io/projected/0cbd0206-072c-4a1f-8cec-1de48f6819c1-kube-api-access-vlzp8\") pod \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") "
Nov 03 10:31:20.124047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.123964    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-config\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-config\") pod \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") "
Nov 03 10:31:20.124047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.123994    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-serving-cert\") pod \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") "
Nov 03 10:31:20.124047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124016    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-trusted-ca-bundle\") pod \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") "
Nov 03 10:31:20.124302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124066    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-oauth-serving-cert\") pod \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") "
Nov 03 10:31:20.124302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124130    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-service-ca\") pod \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\" (UID: \"0cbd0206-072c-4a1f-8cec-1de48f6819c1\") "
Nov 03 10:31:20.124302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124154    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbd0206-072c-4a1f-8cec-1de48f6819c1-tmp" pod "0cbd0206-072c-4a1f-8cec-1de48f6819c1" (UID: "0cbd0206-072c-4a1f-8cec-1de48f6819c1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:31:20.124451 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124350    5393 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbd0206-072c-4a1f-8cec-1de48f6819c1-tmp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:31:20.124508 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124447    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-config" pod "0cbd0206-072c-4a1f-8cec-1de48f6819c1" (UID: "0cbd0206-072c-4a1f-8cec-1de48f6819c1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:31:20.124508 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124486    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-trusted-ca-bundle" pod "0cbd0206-072c-4a1f-8cec-1de48f6819c1" (UID: "0cbd0206-072c-4a1f-8cec-1de48f6819c1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:31:20.124771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124502    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-oauth-serving-cert" pod "0cbd0206-072c-4a1f-8cec-1de48f6819c1" (UID: "0cbd0206-072c-4a1f-8cec-1de48f6819c1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:31:20.124825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.124703    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-service-ca" pod "0cbd0206-072c-4a1f-8cec-1de48f6819c1" (UID: "0cbd0206-072c-4a1f-8cec-1de48f6819c1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:31:20.126230 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.126197    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbd0206-072c-4a1f-8cec-1de48f6819c1-kube-api-access-vlzp8" pod "0cbd0206-072c-4a1f-8cec-1de48f6819c1" (UID: "0cbd0206-072c-4a1f-8cec-1de48f6819c1"). InnerVolumeSpecName "kube-api-access-vlzp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:31:20.126304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.126262    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-oauth-config" pod "0cbd0206-072c-4a1f-8cec-1de48f6819c1" (UID: "0cbd0206-072c-4a1f-8cec-1de48f6819c1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:31:20.126304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.126279    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-serving-cert" pod "0cbd0206-072c-4a1f-8cec-1de48f6819c1" (UID: "0cbd0206-072c-4a1f-8cec-1de48f6819c1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:31:20.224712 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.224680    5393 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:31:20.224712 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.224711    5393 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-oauth-config\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:31:20.224908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.224724    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzp8\" (UniqueName: \"kubernetes.io/projected/0cbd0206-072c-4a1f-8cec-1de48f6819c1-kube-api-access-vlzp8\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:31:20.224908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.224738    5393 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-config\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:31:20.224908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.224750    5393 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cbd0206-072c-4a1f-8cec-1de48f6819c1-console-serving-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:31:20.224908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.224761    5393 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-trusted-ca-bundle\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:31:20.224908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.224773    5393 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cbd0206-072c-4a1f-8cec-1de48f6819c1-oauth-serving-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:31:20.768092 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.768060    5393 generic.go:356] "Generic (PLEG): container finished" podID="0cbd0206-072c-4a1f-8cec-1de48f6819c1" containerID="b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c" exitCode=0
Nov 03 10:31:20.768483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.768122    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d466b654-9zg4p"
Nov 03 10:31:20.768483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.768144    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d466b654-9zg4p" event={"ID":"0cbd0206-072c-4a1f-8cec-1de48f6819c1","Type":"ContainerDied","Data":"b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c"}
Nov 03 10:31:20.768483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.768198    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d466b654-9zg4p" event={"ID":"0cbd0206-072c-4a1f-8cec-1de48f6819c1","Type":"ContainerDied","Data":"20730269d32a7da99e953384b5f7921536c412bf68ebe8be3b1167e0406af4f7"}
Nov 03 10:31:20.768483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.768220    5393 scope.go:122] "RemoveContainer" containerID="b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c"
Nov 03 10:31:20.776095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.776078    5393 scope.go:122] "RemoveContainer" containerID="b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c"
Nov 03 10:31:20.776363 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:31:20.776346    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c\": container with ID starting with b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c not found: ID does not exist" containerID="b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c"
Nov 03 10:31:20.776413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.776373    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c"} err="failed to get container status \"b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c\": rpc error: code = NotFound desc = could not find container \"b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c\": container with ID starting with b764f25b16ab947eea48f57b2bd37d048d6fc6d7c7bc2eefc126ec1e2ef6a11c not found: ID does not exist"
Nov 03 10:31:20.783872 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.783852    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d466b654-9zg4p"]
Nov 03 10:31:20.786801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:20.786781    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67d466b654-9zg4p"]
Nov 03 10:31:22.231239 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:31:22.231199    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbd0206-072c-4a1f-8cec-1de48f6819c1" path="/var/lib/kubelet/pods/0cbd0206-072c-4a1f-8cec-1de48f6819c1/volumes"
Nov 03 10:31:43.187087 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:31:43.187049    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:32:13.187040 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:32:13.187005    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:32:43.187248 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:32:43.187207    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:33:13.187103 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:33:13.187053    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:33:32.971513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.971475    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"]
Nov 03 10:33:32.972141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.971932    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0cbd0206-072c-4a1f-8cec-1de48f6819c1" containerName="console"
Nov 03 10:33:32.972141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.971950    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbd0206-072c-4a1f-8cec-1de48f6819c1" containerName="console"
Nov 03 10:33:32.972141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.972020    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbd0206-072c-4a1f-8cec-1de48f6819c1" containerName="console"
Nov 03 10:33:32.975133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.975111    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:32.986148 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.986122    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 10:33:32.986148 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.986128    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 10:33:32.986283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.986152    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8knf5\""
Nov 03 10:33:32.995567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:32.995540    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"]
Nov 03 10:33:33.003029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.002986    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.003139 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.003057    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.003188 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.003138    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwkzx\" (UniqueName: \"kubernetes.io/projected/294badcc-c1ca-4e3a-9d9d-c08e5386362d-kube-api-access-hwkzx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.104226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.104179    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.104226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.104227    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwkzx\" (UniqueName: \"kubernetes.io/projected/294badcc-c1ca-4e3a-9d9d-c08e5386362d-kube-api-access-hwkzx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.104423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.104279    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.104647 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.104625    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.104693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.104673    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.112857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.112833    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwkzx\" (UniqueName: \"kubernetes.io/projected/294badcc-c1ca-4e3a-9d9d-c08e5386362d-kube-api-access-hwkzx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.279005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.278936    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"]
Nov 03 10:33:33.282274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.282259    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.289330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.289309    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:33.297813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.297787    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 10:33:33.297813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.297798    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 10:33:33.298007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.297800    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 10:33:33.298007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.297985    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 10:33:33.298111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.298024    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 10:33:33.298230 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.298217    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 10:33:33.298307 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.298293    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fbzlj\""
Nov 03 10:33:33.301956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.301937    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"]
Nov 03 10:33:33.305768 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.305748    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.305875 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.305785    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndl8x\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-kube-api-access-ndl8x\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.305875 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.305833    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e13138a-c099-4244-a9e8-d28cb5b7d41e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.306048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.305964    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.306048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.306009    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.306147 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.306056    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.306147 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.306092    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.407205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.407167    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e13138a-c099-4244-a9e8-d28cb5b7d41e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.407205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.407210    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.407442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.407230    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.407442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.407252    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.407442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.407283    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.407442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.407346    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.407442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.407369    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndl8x\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-kube-api-access-ndl8x\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.407895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.407862    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.409791 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.409764    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.409878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.409822    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e13138a-c099-4244-a9e8-d28cb5b7d41e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.409946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.409870    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.409946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.409904    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.415255 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.415235    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.415708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.415691    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndl8x\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-kube-api-access-ndl8x\") pod \"istiod-openshift-gateway-7876bfcb74-gbxpn\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.564386 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.564309    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"]
Nov 03 10:33:33.567572 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:33:33.567541    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294badcc_c1ca_4e3a_9d9d_c08e5386362d.slice/crio-7fc435d3d19fda269d7f624f1eddd91895c9fc2ba6fd2145d658cfead8e513f8 WatchSource:0}: Error finding container 7fc435d3d19fda269d7f624f1eddd91895c9fc2ba6fd2145d658cfead8e513f8: Status 404 returned error can't find the container with id 7fc435d3d19fda269d7f624f1eddd91895c9fc2ba6fd2145d658cfead8e513f8
Nov 03 10:33:33.593373 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.593341    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:33.870582 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:33.870548    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"]
Nov 03 10:33:33.874406 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:33:33.874381    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e13138a_c099_4244_a9e8_d28cb5b7d41e.slice/crio-1d457e6a4a34d453ebb34a12da45de2b795a79ae54f9a06b0ee6658b43c86a1d WatchSource:0}: Error finding container 1d457e6a4a34d453ebb34a12da45de2b795a79ae54f9a06b0ee6658b43c86a1d: Status 404 returned error can't find the container with id 1d457e6a4a34d453ebb34a12da45de2b795a79ae54f9a06b0ee6658b43c86a1d
Nov 03 10:33:34.221599 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:34.221513    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc" event={"ID":"294badcc-c1ca-4e3a-9d9d-c08e5386362d","Type":"ContainerStarted","Data":"7fc435d3d19fda269d7f624f1eddd91895c9fc2ba6fd2145d658cfead8e513f8"}
Nov 03 10:33:34.233377 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:34.233342    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn" event={"ID":"7e13138a-c099-4244-a9e8-d28cb5b7d41e","Type":"ContainerStarted","Data":"1d457e6a4a34d453ebb34a12da45de2b795a79ae54f9a06b0ee6658b43c86a1d"}
Nov 03 10:33:37.005688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:37.005312    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:33:37.005688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:37.005370    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:33:37.236410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:37.236344    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn" event={"ID":"7e13138a-c099-4244-a9e8-d28cb5b7d41e","Type":"ContainerStarted","Data":"c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf"}
Nov 03 10:33:37.236583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:37.236514    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:37.256851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:37.256764    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn" podStartSLOduration=1.128662724 podStartE2EDuration="4.256752653s" totalImagesPullingTime="3.128089929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:33:37.255068322 +0000 UTC m=+261.159922804" watchObservedRunningTime="2026-11-03 10:33:37.256752653 +0000 UTC m=+261.161607090"
Nov 03 10:33:38.241640 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:38.241595    5393 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-gbxpn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:33:38.242037 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:38.241667    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn" podUID="7e13138a-c099-4244-a9e8-d28cb5b7d41e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:33:41.241014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:41.240985    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:33:41.257967 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:41.257911    5393 generic.go:356] "Generic (PLEG): container finished" podID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerID="f59c3589142cd5121fb430e887b1b604f05a0b22fd3b46e5f9d1c946776b8d28" exitCode=0
Nov 03 10:33:41.258134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:41.257972    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc" event={"ID":"294badcc-c1ca-4e3a-9d9d-c08e5386362d","Type":"ContainerDied","Data":"f59c3589142cd5121fb430e887b1b604f05a0b22fd3b46e5f9d1c946776b8d28"}
Nov 03 10:33:43.187438 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:33:43.187403    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:33:45.274827 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:45.274792    5393 generic.go:356] "Generic (PLEG): container finished" podID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerID="fa6b27a747f3cb669514396854cc1f81bddc5ad6c014c5765eca506c1955b37c" exitCode=0
Nov 03 10:33:45.275265 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:45.274851    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc" event={"ID":"294badcc-c1ca-4e3a-9d9d-c08e5386362d","Type":"ContainerDied","Data":"fa6b27a747f3cb669514396854cc1f81bddc5ad6c014c5765eca506c1955b37c"}
Nov 03 10:33:52.305455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:52.305423    5393 generic.go:356] "Generic (PLEG): container finished" podID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerID="5d2d78c9623cda21480020fdef69b2071d214e411d1f0cb6f8090c3427ffce09" exitCode=0
Nov 03 10:33:52.305866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:52.305472    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc" event={"ID":"294badcc-c1ca-4e3a-9d9d-c08e5386362d","Type":"ContainerDied","Data":"5d2d78c9623cda21480020fdef69b2071d214e411d1f0cb6f8090c3427ffce09"}
Nov 03 10:33:53.430182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.430155    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:53.587498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.587398    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294badcc-c1ca-4e3a-9d9d-c08e5386362d-kube-api-access-hwkzx\" (UniqueName: \"kubernetes.io/projected/294badcc-c1ca-4e3a-9d9d-c08e5386362d-kube-api-access-hwkzx\") pod \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") "
Nov 03 10:33:53.587498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.587486    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-util\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-util\") pod \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") "
Nov 03 10:33:53.587688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.587552    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-bundle\") pod \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\" (UID: \"294badcc-c1ca-4e3a-9d9d-c08e5386362d\") "
Nov 03 10:33:53.588247 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.588212    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-bundle" pod "294badcc-c1ca-4e3a-9d9d-c08e5386362d" (UID: "294badcc-c1ca-4e3a-9d9d-c08e5386362d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:33:53.589747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.589728    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294badcc-c1ca-4e3a-9d9d-c08e5386362d-kube-api-access-hwkzx" pod "294badcc-c1ca-4e3a-9d9d-c08e5386362d" (UID: "294badcc-c1ca-4e3a-9d9d-c08e5386362d"). InnerVolumeSpecName "kube-api-access-hwkzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:33:53.591853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.591829    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-util" pod "294badcc-c1ca-4e3a-9d9d-c08e5386362d" (UID: "294badcc-c1ca-4e3a-9d9d-c08e5386362d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:33:53.688646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.688601    5393 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-util\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:33:53.688646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.688637    5393 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/294badcc-c1ca-4e3a-9d9d-c08e5386362d-bundle\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:33:53.688646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:53.688652    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwkzx\" (UniqueName: \"kubernetes.io/projected/294badcc-c1ca-4e3a-9d9d-c08e5386362d-kube-api-access-hwkzx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:33:54.314116 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:54.314079    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc" event={"ID":"294badcc-c1ca-4e3a-9d9d-c08e5386362d","Type":"ContainerDied","Data":"7fc435d3d19fda269d7f624f1eddd91895c9fc2ba6fd2145d658cfead8e513f8"}
Nov 03 10:33:54.314282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:54.314132    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7vcpc"
Nov 03 10:33:54.314282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:33:54.314134    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fc435d3d19fda269d7f624f1eddd91895c9fc2ba6fd2145d658cfead8e513f8"
Nov 03 10:34:13.187442 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:34:13.187405    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:34:16.264576 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:16.264549    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:34:16.266492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:16.266470    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:34:33.101145 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.101055    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"]
Nov 03 10:34:33.101719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.101509    5393 cpu_manager.go:414] "containerMap: removing container" podUID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerName="extract"
Nov 03 10:34:33.101719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.101526    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerName="extract"
Nov 03 10:34:33.101719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.101587    5393 cpu_manager.go:414] "containerMap: removing container" podUID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerName="util"
Nov 03 10:34:33.101719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.101596    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerName="util"
Nov 03 10:34:33.101719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.101608    5393 cpu_manager.go:414] "containerMap: removing container" podUID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerName="pull"
Nov 03 10:34:33.101719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.101616    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerName="pull"
Nov 03 10:34:33.101719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.101694    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="294badcc-c1ca-4e3a-9d9d-c08e5386362d" containerName="extract"
Nov 03 10:34:33.103657 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.103636    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.130468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.130443    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"]
Nov 03 10:34:33.188396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.188364    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 10:34:33.188781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.188763    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 10:34:33.188878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.188837    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-s78mx\""
Nov 03 10:34:33.188878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.188860    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 10:34:33.188878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.188868    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 10:34:33.189022 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.188890    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 10:34:33.192185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.192165    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5311b21b-d51c-4b56-ad10-0829f5842b90-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.192231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.192201    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5txw\" (UniqueName: \"kubernetes.io/projected/5311b21b-d51c-4b56-ad10-0829f5842b90-kube-api-access-b5txw\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.192264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.192232    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.192311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.192295    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5311b21b-d51c-4b56-ad10-0829f5842b90-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.192383 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.192365    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.192416 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.192406    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.293064 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.293022    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5311b21b-d51c-4b56-ad10-0829f5842b90-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.293258 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.293074    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5txw\" (UniqueName: \"kubernetes.io/projected/5311b21b-d51c-4b56-ad10-0829f5842b90-kube-api-access-b5txw\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.293258 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.293102    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.293258 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.293136    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5311b21b-d51c-4b56-ad10-0829f5842b90-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.293258 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.293164    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.293258 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.293212    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.293874 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.293784    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5311b21b-d51c-4b56-ad10-0829f5842b90-manifests\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.295724 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.295698    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5311b21b-d51c-4b56-ad10-0829f5842b90-tmp\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.295849 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.295738    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-webhook-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.295849 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.295806    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-metrics-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.295968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.295944    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5311b21b-d51c-4b56-ad10-0829f5842b90-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.304092 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.304066    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5txw\" (UniqueName: \"kubernetes.io/projected/5311b21b-d51c-4b56-ad10-0829f5842b90-kube-api-access-b5txw\") pod \"opendatahub-operator-controller-manager-6688c64986-t7jsc\" (UID: \"5311b21b-d51c-4b56-ad10-0829f5842b90\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.416018 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.415932    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:33.720695 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:34:33.720659    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5311b21b_d51c_4b56_ad10_0829f5842b90.slice/crio-4c83048afc110a7cd5cb70f75d9cd980dd28464edd50048602b8050b622d55a2 WatchSource:0}: Error finding container 4c83048afc110a7cd5cb70f75d9cd980dd28464edd50048602b8050b622d55a2: Status 404 returned error can't find the container with id 4c83048afc110a7cd5cb70f75d9cd980dd28464edd50048602b8050b622d55a2
Nov 03 10:34:33.722360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:33.722338    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"]
Nov 03 10:34:34.459168 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:34.459124    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc" event={"ID":"5311b21b-d51c-4b56-ad10-0829f5842b90","Type":"ContainerStarted","Data":"4c83048afc110a7cd5cb70f75d9cd980dd28464edd50048602b8050b622d55a2"}
Nov 03 10:34:37.473410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:37.473373    5393 generic.go:356] "Generic (PLEG): container finished" podID="5311b21b-d51c-4b56-ad10-0829f5842b90" containerID="a0040c88e04b9d6a83c8d3fe656a946584ac520d045787be59ef2d78cc972d31" exitCode=0
Nov 03 10:34:37.473787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:37.473437    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc" event={"ID":"5311b21b-d51c-4b56-ad10-0829f5842b90","Type":"ContainerDied","Data":"a0040c88e04b9d6a83c8d3fe656a946584ac520d045787be59ef2d78cc972d31"}
Nov 03 10:34:38.478886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:38.478843    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc" event={"ID":"5311b21b-d51c-4b56-ad10-0829f5842b90","Type":"ContainerStarted","Data":"a244f965754e8174eafa20edfc171fd97a508b696a62f9470479a1919a53eec2"}
Nov 03 10:34:38.479391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:38.478957    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:38.518346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:38.518291    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc" podStartSLOduration=1.618666256 podStartE2EDuration="5.51827699s" totalImagesPullingTime="3.325567848s" totalInitContainerRuntime="574.042886ms" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:38.515902248 +0000 UTC m=+322.420756733" watchObservedRunningTime="2026-11-03 10:34:38.51827699 +0000 UTC m=+322.423131430"
Nov 03 10:34:43.187805 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:34:43.187766    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:34:49.485265 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:49.485234    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6688c64986-t7jsc"
Nov 03 10:34:52.254094 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.254060    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"]
Nov 03 10:34:52.260185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.260166    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"
Nov 03 10:34:52.264357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.264326    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 10:34:52.265647 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.265631    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 10:34:52.265739 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.265720    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k6cwk\""
Nov 03 10:34:52.272504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.272481    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"]
Nov 03 10:34:52.363155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.363122    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32733a19-4409-4789-9c20-4d7e6f727037-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-q57nl\" (UID: \"32733a19-4409-4789-9c20-4d7e6f727037\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"
Nov 03 10:34:52.363312 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.363184    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgdx6\" (UniqueName: \"kubernetes.io/projected/32733a19-4409-4789-9c20-4d7e6f727037-kube-api-access-qgdx6\") pod \"cert-manager-cainjector-5c57bbd4fc-q57nl\" (UID: \"32733a19-4409-4789-9c20-4d7e6f727037\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"
Nov 03 10:34:52.463786 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.463742    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32733a19-4409-4789-9c20-4d7e6f727037-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-q57nl\" (UID: \"32733a19-4409-4789-9c20-4d7e6f727037\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"
Nov 03 10:34:52.463984 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.463808    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgdx6\" (UniqueName: \"kubernetes.io/projected/32733a19-4409-4789-9c20-4d7e6f727037-kube-api-access-qgdx6\") pod \"cert-manager-cainjector-5c57bbd4fc-q57nl\" (UID: \"32733a19-4409-4789-9c20-4d7e6f727037\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"
Nov 03 10:34:52.472762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.472731    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgdx6\" (UniqueName: \"kubernetes.io/projected/32733a19-4409-4789-9c20-4d7e6f727037-kube-api-access-qgdx6\") pod \"cert-manager-cainjector-5c57bbd4fc-q57nl\" (UID: \"32733a19-4409-4789-9c20-4d7e6f727037\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"
Nov 03 10:34:52.472762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.472741    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32733a19-4409-4789-9c20-4d7e6f727037-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-q57nl\" (UID: \"32733a19-4409-4789-9c20-4d7e6f727037\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"
Nov 03 10:34:52.578187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.578103    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"
Nov 03 10:34:52.852355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.852328    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl"]
Nov 03 10:34:52.855450 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:34:52.855423    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32733a19_4409_4789_9c20_4d7e6f727037.slice/crio-dfeacfcb03463e291e3859ee327eef11c8231eef469febf76b9be367f8450277 WatchSource:0}: Error finding container dfeacfcb03463e291e3859ee327eef11c8231eef469febf76b9be367f8450277: Status 404 returned error can't find the container with id dfeacfcb03463e291e3859ee327eef11c8231eef469febf76b9be367f8450277
Nov 03 10:34:52.857356 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:52.857337    5393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:34:53.540728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:53.540696    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl" event={"ID":"32733a19-4409-4789-9c20-4d7e6f727037","Type":"ContainerStarted","Data":"dfeacfcb03463e291e3859ee327eef11c8231eef469febf76b9be367f8450277"}
Nov 03 10:34:56.554042 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:56.554004    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl" event={"ID":"32733a19-4409-4789-9c20-4d7e6f727037","Type":"ContainerStarted","Data":"12d0331132565929576a2c144b16a21d989b750cb64ae9b8887db04413fe1a67"}
Nov 03 10:34:56.574785 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:34:56.574737    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-q57nl" podStartSLOduration=1.878656532 podStartE2EDuration="4.57472336s" totalImagesPullingTime="2.696066828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:34:56.572707384 +0000 UTC m=+340.477561825" watchObservedRunningTime="2026-11-03 10:34:56.57472336 +0000 UTC m=+340.479577798"
Nov 03 10:35:00.150607 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.150574    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-m8lgd"]
Nov 03 10:35:00.153766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.153749    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-m8lgd"
Nov 03 10:35:00.163024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.162998    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9lpwp\""
Nov 03 10:35:00.164493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.164472    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-m8lgd"]
Nov 03 10:35:00.233058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.233028    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9qh\" (UniqueName: \"kubernetes.io/projected/1353ba67-0cb9-48df-9742-090851aadafb-kube-api-access-7f9qh\") pod \"cert-manager-597b95cb84-m8lgd\" (UID: \"1353ba67-0cb9-48df-9742-090851aadafb\") " pod="cert-manager/cert-manager-597b95cb84-m8lgd"
Nov 03 10:35:00.233222 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.233099    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1353ba67-0cb9-48df-9742-090851aadafb-bound-sa-token\") pod \"cert-manager-597b95cb84-m8lgd\" (UID: \"1353ba67-0cb9-48df-9742-090851aadafb\") " pod="cert-manager/cert-manager-597b95cb84-m8lgd"
Nov 03 10:35:00.334473 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.334440    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1353ba67-0cb9-48df-9742-090851aadafb-bound-sa-token\") pod \"cert-manager-597b95cb84-m8lgd\" (UID: \"1353ba67-0cb9-48df-9742-090851aadafb\") " pod="cert-manager/cert-manager-597b95cb84-m8lgd"
Nov 03 10:35:00.334580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.334502    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9qh\" (UniqueName: \"kubernetes.io/projected/1353ba67-0cb9-48df-9742-090851aadafb-kube-api-access-7f9qh\") pod \"cert-manager-597b95cb84-m8lgd\" (UID: \"1353ba67-0cb9-48df-9742-090851aadafb\") " pod="cert-manager/cert-manager-597b95cb84-m8lgd"
Nov 03 10:35:00.342639 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.342604    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1353ba67-0cb9-48df-9742-090851aadafb-bound-sa-token\") pod \"cert-manager-597b95cb84-m8lgd\" (UID: \"1353ba67-0cb9-48df-9742-090851aadafb\") " pod="cert-manager/cert-manager-597b95cb84-m8lgd"
Nov 03 10:35:00.342757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.342678    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9qh\" (UniqueName: \"kubernetes.io/projected/1353ba67-0cb9-48df-9742-090851aadafb-kube-api-access-7f9qh\") pod \"cert-manager-597b95cb84-m8lgd\" (UID: \"1353ba67-0cb9-48df-9742-090851aadafb\") " pod="cert-manager/cert-manager-597b95cb84-m8lgd"
Nov 03 10:35:00.465680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.465592    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-m8lgd"
Nov 03 10:35:00.738583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:00.738391    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-m8lgd"]
Nov 03 10:35:00.741275 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:35:00.741241    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1353ba67_0cb9_48df_9742_090851aadafb.slice/crio-9ff81d77c97c54819154790d42114de210a1cc1ba9bbc20ee11a7ad974fbb83d WatchSource:0}: Error finding container 9ff81d77c97c54819154790d42114de210a1cc1ba9bbc20ee11a7ad974fbb83d: Status 404 returned error can't find the container with id 9ff81d77c97c54819154790d42114de210a1cc1ba9bbc20ee11a7ad974fbb83d
Nov 03 10:35:01.576934 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:01.576889    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-m8lgd" event={"ID":"1353ba67-0cb9-48df-9742-090851aadafb","Type":"ContainerStarted","Data":"dae5e58dfa1f47b06854882ae6913a3736757b962fb54e49efc5b94f8301eec4"}
Nov 03 10:35:01.577408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:01.576957    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-m8lgd" event={"ID":"1353ba67-0cb9-48df-9742-090851aadafb","Type":"ContainerStarted","Data":"9ff81d77c97c54819154790d42114de210a1cc1ba9bbc20ee11a7ad974fbb83d"}
Nov 03 10:35:01.593100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:01.593055    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-m8lgd" podStartSLOduration=1.593041972 podStartE2EDuration="1.593041972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:01.592474909 +0000 UTC m=+345.497329351" watchObservedRunningTime="2026-11-03 10:35:01.593041972 +0000 UTC m=+345.497896409"
Nov 03 10:35:04.959542 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:04.959510    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"]
Nov 03 10:35:04.965758 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:04.965742    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:04.977409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:04.977381    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"]
Nov 03 10:35:04.987802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:04.987774    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 10:35:04.988045 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:04.988027    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 10:35:04.988145 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:04.988081    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5xvj2\""
Nov 03 10:35:05.074636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.074600    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktkk\" (UniqueName: \"kubernetes.io/projected/e3236943-9a1b-4a62-8992-fd40416c20df-kube-api-access-4ktkk\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.074861 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.074644    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3236943-9a1b-4a62-8992-fd40416c20df-tmp\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.074861 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.074791    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3236943-9a1b-4a62-8992-fd40416c20df-tls-certs\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.176300 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.176258    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktkk\" (UniqueName: \"kubernetes.io/projected/e3236943-9a1b-4a62-8992-fd40416c20df-kube-api-access-4ktkk\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.176300 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.176305    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3236943-9a1b-4a62-8992-fd40416c20df-tmp\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.176562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.176407    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3236943-9a1b-4a62-8992-fd40416c20df-tls-certs\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.178824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.178795    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3236943-9a1b-4a62-8992-fd40416c20df-tmp\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.178988 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.178969    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3236943-9a1b-4a62-8992-fd40416c20df-tls-certs\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.185288 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.185260    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktkk\" (UniqueName: \"kubernetes.io/projected/e3236943-9a1b-4a62-8992-fd40416c20df-kube-api-access-4ktkk\") pod \"kube-auth-proxy-5cb777df86-kcgc4\" (UID: \"e3236943-9a1b-4a62-8992-fd40416c20df\") " pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.277972 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.277870    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"
Nov 03 10:35:05.554436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.554415    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4"]
Nov 03 10:35:05.556748 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:35:05.556719    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3236943_9a1b_4a62_8992_fd40416c20df.slice/crio-e629311cbdfcf133407d4563b0989b035daff8d9199298de1bcc7635c387ba5e WatchSource:0}: Error finding container e629311cbdfcf133407d4563b0989b035daff8d9199298de1bcc7635c387ba5e: Status 404 returned error can't find the container with id e629311cbdfcf133407d4563b0989b035daff8d9199298de1bcc7635c387ba5e
Nov 03 10:35:05.593681 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:05.593651    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4" event={"ID":"e3236943-9a1b-4a62-8992-fd40416c20df","Type":"ContainerStarted","Data":"e629311cbdfcf133407d4563b0989b035daff8d9199298de1bcc7635c387ba5e"}
Nov 03 10:35:08.608453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:08.608358    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4" event={"ID":"e3236943-9a1b-4a62-8992-fd40416c20df","Type":"ContainerStarted","Data":"2098d2418b37ea199172b246d24799413981c86133f838bd9e3efc81f1739d53"}
Nov 03 10:35:08.624216 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:08.624165    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cb777df86-kcgc4" podStartSLOduration=1.830473837 podStartE2EDuration="4.62414977s" totalImagesPullingTime="2.793675933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:08.624021008 +0000 UTC m=+352.528875449" watchObservedRunningTime="2026-11-03 10:35:08.62414977 +0000 UTC m=+352.529004211"
Nov 03 10:35:13.187863 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:35:13.187828    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:35:35.040030 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.039996    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"]
Nov 03 10:35:35.043329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.043307    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:35.052065 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.052020    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 10:35:35.053395 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.053219    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 10:35:35.053395 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.053326    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xpkj5\""
Nov 03 10:35:35.053395 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.053363    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 10:35:35.054415 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.054396    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"]
Nov 03 10:35:35.120854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.120818    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8zd\" (UniqueName: \"kubernetes.io/projected/20a711ee-b065-40f4-9b77-fabc6da0dc85-kube-api-access-lm8zd\") pod \"limitador-limitador-74dfcfb894-5kwcd\" (UID: \"20a711ee-b065-40f4-9b77-fabc6da0dc85\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:35.121043 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.120868    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20a711ee-b065-40f4-9b77-fabc6da0dc85-config-file\") pod \"limitador-limitador-74dfcfb894-5kwcd\" (UID: \"20a711ee-b065-40f4-9b77-fabc6da0dc85\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:35.148163 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.148130    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"]
Nov 03 10:35:35.222049 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.222017    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8zd\" (UniqueName: \"kubernetes.io/projected/20a711ee-b065-40f4-9b77-fabc6da0dc85-kube-api-access-lm8zd\") pod \"limitador-limitador-74dfcfb894-5kwcd\" (UID: \"20a711ee-b065-40f4-9b77-fabc6da0dc85\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:35.222185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.222067    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20a711ee-b065-40f4-9b77-fabc6da0dc85-config-file\") pod \"limitador-limitador-74dfcfb894-5kwcd\" (UID: \"20a711ee-b065-40f4-9b77-fabc6da0dc85\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:35.222676 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.222657    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20a711ee-b065-40f4-9b77-fabc6da0dc85-config-file\") pod \"limitador-limitador-74dfcfb894-5kwcd\" (UID: \"20a711ee-b065-40f4-9b77-fabc6da0dc85\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:35.230453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.230426    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8zd\" (UniqueName: \"kubernetes.io/projected/20a711ee-b065-40f4-9b77-fabc6da0dc85-kube-api-access-lm8zd\") pod \"limitador-limitador-74dfcfb894-5kwcd\" (UID: \"20a711ee-b065-40f4-9b77-fabc6da0dc85\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:35.356293 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.356263    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:35.636382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.636355    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"]
Nov 03 10:35:35.639055 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:35:35.639027    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a711ee_b065_40f4_9b77_fabc6da0dc85.slice/crio-18b477d9607ce71574382314ddd9c6964a8d9367e40fca9a408b51abcabc9ceb WatchSource:0}: Error finding container 18b477d9607ce71574382314ddd9c6964a8d9367e40fca9a408b51abcabc9ceb: Status 404 returned error can't find the container with id 18b477d9607ce71574382314ddd9c6964a8d9367e40fca9a408b51abcabc9ceb
Nov 03 10:35:35.722228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:35.722176    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd" event={"ID":"20a711ee-b065-40f4-9b77-fabc6da0dc85","Type":"ContainerStarted","Data":"18b477d9607ce71574382314ddd9c6964a8d9367e40fca9a408b51abcabc9ceb"}
Nov 03 10:35:38.737478 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:38.737390    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd" event={"ID":"20a711ee-b065-40f4-9b77-fabc6da0dc85","Type":"ContainerStarted","Data":"a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845"}
Nov 03 10:35:38.737899 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:38.737539    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:38.757393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:38.757346    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd" podStartSLOduration=0.959193188 podStartE2EDuration="3.757329873s" totalImagesPullingTime="2.798136685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:38.756559104 +0000 UTC m=+382.661413567" watchObservedRunningTime="2026-11-03 10:35:38.757329873 +0000 UTC m=+382.662184313"
Nov 03 10:35:43.191252 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:35:43.191211    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:35:48.813068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:48.813028    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"]
Nov 03 10:35:48.813492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:48.813233    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd" podUID="20a711ee-b065-40f4-9b77-fabc6da0dc85" containerName="limitador" containerID="cri-o://a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845" gracePeriod=30
Nov 03 10:35:48.814038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:48.813958    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:49.756360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.756327    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:49.782208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.782176    5393 generic.go:356] "Generic (PLEG): container finished" podID="20a711ee-b065-40f4-9b77-fabc6da0dc85" containerID="a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845" exitCode=0
Nov 03 10:35:49.782353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.782216    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"
Nov 03 10:35:49.782353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.782253    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd" event={"ID":"20a711ee-b065-40f4-9b77-fabc6da0dc85","Type":"ContainerDied","Data":"a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845"}
Nov 03 10:35:49.782353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.782309    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-5kwcd" event={"ID":"20a711ee-b065-40f4-9b77-fabc6da0dc85","Type":"ContainerDied","Data":"18b477d9607ce71574382314ddd9c6964a8d9367e40fca9a408b51abcabc9ceb"}
Nov 03 10:35:49.782353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.782332    5393 scope.go:122] "RemoveContainer" containerID="a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845"
Nov 03 10:35:49.791984 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.791957    5393 scope.go:122] "RemoveContainer" containerID="a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845"
Nov 03 10:35:49.792285 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:35:49.792267    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845\": container with ID starting with a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845 not found: ID does not exist" containerID="a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845"
Nov 03 10:35:49.792350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.792293    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845"} err="failed to get container status \"a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845\": rpc error: code = NotFound desc = could not find container \"a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845\": container with ID starting with a0023dcaa4ca247c0465c8a6647e3fa0c87b0a42086089fe25d8bf9938a20845 not found: ID does not exist"
Nov 03 10:35:49.846557 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.846526    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a711ee-b065-40f4-9b77-fabc6da0dc85-kube-api-access-lm8zd\" (UniqueName: \"kubernetes.io/projected/20a711ee-b065-40f4-9b77-fabc6da0dc85-kube-api-access-lm8zd\") pod \"20a711ee-b065-40f4-9b77-fabc6da0dc85\" (UID: \"20a711ee-b065-40f4-9b77-fabc6da0dc85\") "
Nov 03 10:35:49.847051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.846606    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a711ee-b065-40f4-9b77-fabc6da0dc85-config-file\" (UniqueName: \"kubernetes.io/configmap/20a711ee-b065-40f4-9b77-fabc6da0dc85-config-file\") pod \"20a711ee-b065-40f4-9b77-fabc6da0dc85\" (UID: \"20a711ee-b065-40f4-9b77-fabc6da0dc85\") "
Nov 03 10:35:49.847051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.846948    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a711ee-b065-40f4-9b77-fabc6da0dc85-config-file" pod "20a711ee-b065-40f4-9b77-fabc6da0dc85" (UID: "20a711ee-b065-40f4-9b77-fabc6da0dc85"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:35:49.848856 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.848834    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a711ee-b065-40f4-9b77-fabc6da0dc85-kube-api-access-lm8zd" pod "20a711ee-b065-40f4-9b77-fabc6da0dc85" (UID: "20a711ee-b065-40f4-9b77-fabc6da0dc85"). InnerVolumeSpecName "kube-api-access-lm8zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:35:49.947526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.947430    5393 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20a711ee-b065-40f4-9b77-fabc6da0dc85-config-file\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:35:49.947526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:49.947468    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8zd\" (UniqueName: \"kubernetes.io/projected/20a711ee-b065-40f4-9b77-fabc6da0dc85-kube-api-access-lm8zd\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:35:50.102470 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:50.102439    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"]
Nov 03 10:35:50.108469 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:50.108441    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-5kwcd"]
Nov 03 10:35:50.228523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:50.228421    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a711ee-b065-40f4-9b77-fabc6da0dc85" path="/var/lib/kubelet/pods/20a711ee-b065-40f4-9b77-fabc6da0dc85/volumes"
Nov 03 10:35:53.804845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.804808    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"]
Nov 03 10:35:53.805350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.805314    5393 cpu_manager.go:414] "containerMap: removing container" podUID="20a711ee-b065-40f4-9b77-fabc6da0dc85" containerName="limitador"
Nov 03 10:35:53.805350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.805332    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a711ee-b065-40f4-9b77-fabc6da0dc85" containerName="limitador"
Nov 03 10:35:53.805457 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.805424    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a711ee-b065-40f4-9b77-fabc6da0dc85" containerName="limitador"
Nov 03 10:35:53.810229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.810207    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:53.814707 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.814681    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"]
Nov 03 10:35:53.816308 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.816286    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 10:35:53.817624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.817602    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 10:35:53.887902 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.887859    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 10:35:53.888095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.887979    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2gwtt\""
Nov 03 10:35:53.982363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.982321    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hg4\" (UniqueName: \"kubernetes.io/projected/32224c3b-fe76-434f-8b26-a07d4419b2f8-kube-api-access-72hg4\") pod \"postgres-7cdbf6dc44-dzp47\" (UID: \"32224c3b-fe76-434f-8b26-a07d4419b2f8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:53.982527 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:53.982385    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32224c3b-fe76-434f-8b26-a07d4419b2f8-data\") pod \"postgres-7cdbf6dc44-dzp47\" (UID: \"32224c3b-fe76-434f-8b26-a07d4419b2f8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:54.083348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:54.083270    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hg4\" (UniqueName: \"kubernetes.io/projected/32224c3b-fe76-434f-8b26-a07d4419b2f8-kube-api-access-72hg4\") pod \"postgres-7cdbf6dc44-dzp47\" (UID: \"32224c3b-fe76-434f-8b26-a07d4419b2f8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:54.083348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:54.083325    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32224c3b-fe76-434f-8b26-a07d4419b2f8-data\") pod \"postgres-7cdbf6dc44-dzp47\" (UID: \"32224c3b-fe76-434f-8b26-a07d4419b2f8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:54.083651 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:54.083635    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32224c3b-fe76-434f-8b26-a07d4419b2f8-data\") pod \"postgres-7cdbf6dc44-dzp47\" (UID: \"32224c3b-fe76-434f-8b26-a07d4419b2f8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:54.095105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:54.095074    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hg4\" (UniqueName: \"kubernetes.io/projected/32224c3b-fe76-434f-8b26-a07d4419b2f8-kube-api-access-72hg4\") pod \"postgres-7cdbf6dc44-dzp47\" (UID: \"32224c3b-fe76-434f-8b26-a07d4419b2f8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:54.123882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:54.123850    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:54.394717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:54.394684    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"]
Nov 03 10:35:54.397615 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:35:54.397590    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32224c3b_fe76_434f_8b26_a07d4419b2f8.slice/crio-103416ab3462fe8650754a28c0be1e25e981d363a67fc4b03599a7a535db8b78 WatchSource:0}: Error finding container 103416ab3462fe8650754a28c0be1e25e981d363a67fc4b03599a7a535db8b78: Status 404 returned error can't find the container with id 103416ab3462fe8650754a28c0be1e25e981d363a67fc4b03599a7a535db8b78
Nov 03 10:35:54.805208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:54.805122    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47" event={"ID":"32224c3b-fe76-434f-8b26-a07d4419b2f8","Type":"ContainerStarted","Data":"103416ab3462fe8650754a28c0be1e25e981d363a67fc4b03599a7a535db8b78"}
Nov 03 10:35:59.830315 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:59.830282    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47" event={"ID":"32224c3b-fe76-434f-8b26-a07d4419b2f8","Type":"ContainerStarted","Data":"286e5438d4c36b782b53fbd1ed957fd1ef08360d6c3774983621e6d9d9ccd180"}
Nov 03 10:35:59.830732 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:59.830426    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:35:59.847672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:35:59.847623    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47" podStartSLOduration=1.5653409539999998 podStartE2EDuration="6.847609461s" totalImagesPullingTime="5.282268507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:35:59.846013957 +0000 UTC m=+403.750868410" watchObservedRunningTime="2026-11-03 10:35:59.847609461 +0000 UTC m=+403.752463902"
Nov 03 10:36:01.112994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.112959    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xmpx9"]
Nov 03 10:36:01.118556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.118528    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:01.124178 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.124157    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 10:36:01.124470 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.124452    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dcb2g\""
Nov 03 10:36:01.124808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.124791    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 10:36:01.125059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.125043    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 10:36:01.125278 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.125251    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xmpx9"]
Nov 03 10:36:01.132583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.132557    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"]
Nov 03 10:36:01.135832 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.135804    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:01.145950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.145901    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 10:36:01.146982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.146357    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vcq5q\""
Nov 03 10:36:01.147677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.147656    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"]
Nov 03 10:36:01.232546 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.232512    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"]
Nov 03 10:36:01.235909 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.235888    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.246817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.246788    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7943ffbe-ca5b-4751-a47e-9a06b04473ea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b7xzs\" (UID: \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:01.246949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.246843    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9s64\" (UniqueName: \"kubernetes.io/projected/ff61cb9b-f039-48cf-8879-d73eb8ff0a4a-kube-api-access-s9s64\") pod \"kserve-controller-manager-776cc66d47-xmpx9\" (UID: \"ff61cb9b-f039-48cf-8879-d73eb8ff0a4a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:01.247019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.246954    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh2vd\" (UniqueName: \"kubernetes.io/projected/7943ffbe-ca5b-4751-a47e-9a06b04473ea-kube-api-access-fh2vd\") pod \"llmisvc-controller-manager-869d5dcc5d-b7xzs\" (UID: \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:01.247019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.246985    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff61cb9b-f039-48cf-8879-d73eb8ff0a4a-cert\") pod \"kserve-controller-manager-776cc66d47-xmpx9\" (UID: \"ff61cb9b-f039-48cf-8879-d73eb8ff0a4a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:01.247559 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.247536    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"]
Nov 03 10:36:01.348415 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348376    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.348415 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348417    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.348683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348442    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5877c92-09b4-4dff-ad52-38660d7345fe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.348683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348459    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.348683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348506    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.348683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348644    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh2vd\" (UniqueName: \"kubernetes.io/projected/7943ffbe-ca5b-4751-a47e-9a06b04473ea-kube-api-access-fh2vd\") pod \"llmisvc-controller-manager-869d5dcc5d-b7xzs\" (UID: \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:01.348683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348680    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff61cb9b-f039-48cf-8879-d73eb8ff0a4a-cert\") pod \"kserve-controller-manager-776cc66d47-xmpx9\" (UID: \"ff61cb9b-f039-48cf-8879-d73eb8ff0a4a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:01.348952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348722    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7943ffbe-ca5b-4751-a47e-9a06b04473ea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b7xzs\" (UID: \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:01.348952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348759    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnrr\" (UniqueName: \"kubernetes.io/projected/b5877c92-09b4-4dff-ad52-38660d7345fe-kube-api-access-2fnrr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.348952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348809    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9s64\" (UniqueName: \"kubernetes.io/projected/ff61cb9b-f039-48cf-8879-d73eb8ff0a4a-kube-api-access-s9s64\") pod \"kserve-controller-manager-776cc66d47-xmpx9\" (UID: \"ff61cb9b-f039-48cf-8879-d73eb8ff0a4a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:01.348952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.348869    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.351423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.351395    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff61cb9b-f039-48cf-8879-d73eb8ff0a4a-cert\") pod \"kserve-controller-manager-776cc66d47-xmpx9\" (UID: \"ff61cb9b-f039-48cf-8879-d73eb8ff0a4a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:01.351529 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.351425    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7943ffbe-ca5b-4751-a47e-9a06b04473ea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-b7xzs\" (UID: \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:01.357891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.357871    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9s64\" (UniqueName: \"kubernetes.io/projected/ff61cb9b-f039-48cf-8879-d73eb8ff0a4a-kube-api-access-s9s64\") pod \"kserve-controller-manager-776cc66d47-xmpx9\" (UID: \"ff61cb9b-f039-48cf-8879-d73eb8ff0a4a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:01.358049 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.357966    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh2vd\" (UniqueName: \"kubernetes.io/projected/7943ffbe-ca5b-4751-a47e-9a06b04473ea-kube-api-access-fh2vd\") pod \"llmisvc-controller-manager-869d5dcc5d-b7xzs\" (UID: \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:01.435339 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.435261    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:01.449591 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.449565    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.449686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.449602    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.449686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.449622    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5877c92-09b4-4dff-ad52-38660d7345fe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.449806 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.449763    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.450024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.449813    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.450024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.449935    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnrr\" (UniqueName: \"kubernetes.io/projected/b5877c92-09b4-4dff-ad52-38660d7345fe-kube-api-access-2fnrr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.450024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.449999    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.450177 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.450021    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:01.451208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.450876    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.452641 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.452618    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5877c92-09b4-4dff-ad52-38660d7345fe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.452736 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.452651    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.452736 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.452716    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.453087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.453070    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.473284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.473248    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnrr\" (UniqueName: \"kubernetes.io/projected/b5877c92-09b4-4dff-ad52-38660d7345fe-kube-api-access-2fnrr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.473515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.473493    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5877c92-09b4-4dff-ad52-38660d7345fe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xm7qt\" (UID: \"b5877c92-09b4-4dff-ad52-38660d7345fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.547353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.547319    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:01.720338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.720265    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xmpx9"]
Nov 03 10:36:01.723730 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:36:01.723695    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff61cb9b_f039_48cf_8879_d73eb8ff0a4a.slice/crio-510c3374e79742e4b4df5a3ccf6c564a7619c2434f942443ed7a4c9c38a08979 WatchSource:0}: Error finding container 510c3374e79742e4b4df5a3ccf6c564a7619c2434f942443ed7a4c9c38a08979: Status 404 returned error can't find the container with id 510c3374e79742e4b4df5a3ccf6c564a7619c2434f942443ed7a4c9c38a08979
Nov 03 10:36:01.740340 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.740313    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"]
Nov 03 10:36:01.743237 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:36:01.743210    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7943ffbe_ca5b_4751_a47e_9a06b04473ea.slice/crio-f29482fc378881fd7836a675d52c53e59bb6f6eb1d2166a2125754b0d3866941 WatchSource:0}: Error finding container f29482fc378881fd7836a675d52c53e59bb6f6eb1d2166a2125754b0d3866941: Status 404 returned error can't find the container with id f29482fc378881fd7836a675d52c53e59bb6f6eb1d2166a2125754b0d3866941
Nov 03 10:36:01.826062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.826032    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"]
Nov 03 10:36:01.826859 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:36:01.826834    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5877c92_09b4_4dff_ad52_38660d7345fe.slice/crio-49efbad305462d5aac8e6b2f7fe9cdc52797e46d564b35c8c63e8f9baa50e547 WatchSource:0}: Error finding container 49efbad305462d5aac8e6b2f7fe9cdc52797e46d564b35c8c63e8f9baa50e547: Status 404 returned error can't find the container with id 49efbad305462d5aac8e6b2f7fe9cdc52797e46d564b35c8c63e8f9baa50e547
Nov 03 10:36:01.829096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.829068    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:36:01.829178 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.829128    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:36:01.840176 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.840147    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9" event={"ID":"ff61cb9b-f039-48cf-8879-d73eb8ff0a4a","Type":"ContainerStarted","Data":"510c3374e79742e4b4df5a3ccf6c564a7619c2434f942443ed7a4c9c38a08979"}
Nov 03 10:36:01.841643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.841621    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt" event={"ID":"b5877c92-09b4-4dff-ad52-38660d7345fe","Type":"ContainerStarted","Data":"49efbad305462d5aac8e6b2f7fe9cdc52797e46d564b35c8c63e8f9baa50e547"}
Nov 03 10:36:01.842757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:01.842735    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs" event={"ID":"7943ffbe-ca5b-4751-a47e-9a06b04473ea","Type":"ContainerStarted","Data":"f29482fc378881fd7836a675d52c53e59bb6f6eb1d2166a2125754b0d3866941"}
Nov 03 10:36:02.849893 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:02.849852    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt" event={"ID":"b5877c92-09b4-4dff-ad52-38660d7345fe","Type":"ContainerStarted","Data":"6f20552420b1e2a42a6110a7ba18ccb8701f542ab4cefd34239dca03fa1e6944"}
Nov 03 10:36:02.850374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:02.850102    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:02.852229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:02.852180    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt"
Nov 03 10:36:02.871660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:02.871596    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xm7qt" podStartSLOduration=1.871575768 podStartE2EDuration="1.871575768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:02.868819934 +0000 UTC m=+406.773674376" watchObservedRunningTime="2026-11-03 10:36:02.871575768 +0000 UTC m=+406.776430211"
Nov 03 10:36:02.939408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:02.939370    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"]
Nov 03 10:36:02.939672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:02.939629    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn" podUID="7e13138a-c099-4244-a9e8-d28cb5b7d41e" containerName="discovery" containerID="cri-o://c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf" gracePeriod=30
Nov 03 10:36:03.201043 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.201014    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:36:03.267579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.267550    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-dns-cert\") pod \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") "
Nov 03 10:36:03.267749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.267613    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e13138a-c099-4244-a9e8-d28cb5b7d41e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e13138a-c099-4244-a9e8-d28cb5b7d41e-local-certs\") pod \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") "
Nov 03 10:36:03.267749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.267633    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-kubeconfig\") pod \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") "
Nov 03 10:36:03.267749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.267655    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-kube-api-access-ndl8x\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-kube-api-access-ndl8x\") pod \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") "
Nov 03 10:36:03.267749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.267673    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-ca-configmap\") pod \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") "
Nov 03 10:36:03.267749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.267734    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-cacerts\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-cacerts\") pod \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") "
Nov 03 10:36:03.268036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.267765    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-token\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-token\") pod \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\" (UID: \"7e13138a-c099-4244-a9e8-d28cb5b7d41e\") "
Nov 03 10:36:03.268180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.268142    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-ca-configmap" pod "7e13138a-c099-4244-a9e8-d28cb5b7d41e" (UID: "7e13138a-c099-4244-a9e8-d28cb5b7d41e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:36:03.270640 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.270612    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e13138a-c099-4244-a9e8-d28cb5b7d41e-local-certs" pod "7e13138a-c099-4244-a9e8-d28cb5b7d41e" (UID: "7e13138a-c099-4244-a9e8-d28cb5b7d41e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:36:03.270744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.270676    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-kubeconfig" pod "7e13138a-c099-4244-a9e8-d28cb5b7d41e" (UID: "7e13138a-c099-4244-a9e8-d28cb5b7d41e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:36:03.270744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.270684    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-kube-api-access-ndl8x" pod "7e13138a-c099-4244-a9e8-d28cb5b7d41e" (UID: "7e13138a-c099-4244-a9e8-d28cb5b7d41e"). InnerVolumeSpecName "kube-api-access-ndl8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:36:03.270744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.270695    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-dns-cert" pod "7e13138a-c099-4244-a9e8-d28cb5b7d41e" (UID: "7e13138a-c099-4244-a9e8-d28cb5b7d41e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:36:03.270856 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.270762    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-cacerts" pod "7e13138a-c099-4244-a9e8-d28cb5b7d41e" (UID: "7e13138a-c099-4244-a9e8-d28cb5b7d41e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:36:03.270856 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.270765    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-token" pod "7e13138a-c099-4244-a9e8-d28cb5b7d41e" (UID: "7e13138a-c099-4244-a9e8-d28cb5b7d41e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:36:03.369191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.369109    5393 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-cacerts\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:03.369191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.369142    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:03.369191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.369155    5393 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-dns-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:03.369191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.369169    5393 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e13138a-c099-4244-a9e8-d28cb5b7d41e-local-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:03.369191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.369183    5393 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-kubeconfig\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:03.369467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.369197    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndl8x\" (UniqueName: \"kubernetes.io/projected/7e13138a-c099-4244-a9e8-d28cb5b7d41e-kube-api-access-ndl8x\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:03.369467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.369212    5393 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e13138a-c099-4244-a9e8-d28cb5b7d41e-istio-csr-ca-configmap\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:03.857169 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.857126    5393 generic.go:356] "Generic (PLEG): container finished" podID="7e13138a-c099-4244-a9e8-d28cb5b7d41e" containerID="c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf" exitCode=0
Nov 03 10:36:03.857619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.857561    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"
Nov 03 10:36:03.857908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.857884    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn" event={"ID":"7e13138a-c099-4244-a9e8-d28cb5b7d41e","Type":"ContainerDied","Data":"c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf"}
Nov 03 10:36:03.858054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.857953    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn" event={"ID":"7e13138a-c099-4244-a9e8-d28cb5b7d41e","Type":"ContainerDied","Data":"1d457e6a4a34d453ebb34a12da45de2b795a79ae54f9a06b0ee6658b43c86a1d"}
Nov 03 10:36:03.858054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.857980    5393 scope.go:122] "RemoveContainer" containerID="c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf"
Nov 03 10:36:03.870601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.870546    5393 scope.go:122] "RemoveContainer" containerID="c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf"
Nov 03 10:36:03.871003 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:36:03.870976    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf\": container with ID starting with c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf not found: ID does not exist" containerID="c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf"
Nov 03 10:36:03.871134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.871017    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf"} err="failed to get container status \"c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf\": rpc error: code = NotFound desc = could not find container \"c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf\": container with ID starting with c3e487648170f601947f2322008a6c26f4eca8a0b088a318292fddad7b9811cf not found: ID does not exist"
Nov 03 10:36:03.884208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.884177    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"]
Nov 03 10:36:03.889057 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:03.889030    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-gbxpn"]
Nov 03 10:36:04.228261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:04.228230    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e13138a-c099-4244-a9e8-d28cb5b7d41e" path="/var/lib/kubelet/pods/7e13138a-c099-4244-a9e8-d28cb5b7d41e/volumes"
Nov 03 10:36:04.863439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:04.863403    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs" event={"ID":"7943ffbe-ca5b-4751-a47e-9a06b04473ea","Type":"ContainerStarted","Data":"de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56"}
Nov 03 10:36:04.863933 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:04.863654    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:04.865040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:04.865013    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9" event={"ID":"ff61cb9b-f039-48cf-8879-d73eb8ff0a4a","Type":"ContainerStarted","Data":"30dacde5be777c730aa41863543ff2ae7a2c68231bc215b75080114f97cb875b"}
Nov 03 10:36:04.865177 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:04.865159    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:04.879073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:04.879035    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs" podStartSLOduration=1.587611342 podStartE2EDuration="3.87902356s" totalImagesPullingTime="2.291412218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:04.878498398 +0000 UTC m=+408.783352840" watchObservedRunningTime="2026-11-03 10:36:04.87902356 +0000 UTC m=+408.783877991"
Nov 03 10:36:04.893173 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:04.893131    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9" podStartSLOduration=1.384948154 podStartE2EDuration="3.893118607s" totalImagesPullingTime="2.508170453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:04.892515898 +0000 UTC m=+408.797370343" watchObservedRunningTime="2026-11-03 10:36:04.893118607 +0000 UTC m=+408.797973048"
Nov 03 10:36:05.863310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:05.863280    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dzp47"
Nov 03 10:36:13.043067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.043034    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67566d777-jjbdl"]
Nov 03 10:36:13.043462 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.043389    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7e13138a-c099-4244-a9e8-d28cb5b7d41e" containerName="discovery"
Nov 03 10:36:13.043462 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.043401    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e13138a-c099-4244-a9e8-d28cb5b7d41e" containerName="discovery"
Nov 03 10:36:13.043530 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.043465    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e13138a-c099-4244-a9e8-d28cb5b7d41e" containerName="discovery"
Nov 03 10:36:13.062236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.062205    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67566d777-jjbdl"]
Nov 03 10:36:13.062389 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.062342    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.066515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.066496    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 10:36:13.068013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.067982    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4jtfj\""
Nov 03 10:36:13.068013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.068001    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 10:36:13.068156 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.068067    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 10:36:13.152937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.152891    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-metrics-certs\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.153086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.152966    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhv4\" (UniqueName: \"kubernetes.io/projected/f515dc20-2b5f-48cb-9623-8ea7440187e9-kube-api-access-cvhv4\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.153132 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.153083    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-webhook-cert\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.187765 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:36:13.187732    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:36:13.253794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.253752    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhv4\" (UniqueName: \"kubernetes.io/projected/f515dc20-2b5f-48cb-9623-8ea7440187e9-kube-api-access-cvhv4\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.254059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.253816    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-webhook-cert\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.254059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.253878    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-metrics-certs\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.256458 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.256434    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-webhook-cert\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.256576 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.256506    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-metrics-certs\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.262393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.262368    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhv4\" (UniqueName: \"kubernetes.io/projected/f515dc20-2b5f-48cb-9623-8ea7440187e9-kube-api-access-cvhv4\") pod \"maas-controller-67566d777-jjbdl\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") " pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.375067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.375023    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:13.535291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.535263    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fdb79d597-9n4gw"]
Nov 03 10:36:13.539460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.539440    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.549957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.549931    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fdb79d597-9n4gw"]
Nov 03 10:36:13.646487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.646291    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67566d777-jjbdl"]
Nov 03 10:36:13.649383 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:36:13.649357    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf515dc20_2b5f_48cb_9623_8ea7440187e9.slice/crio-a86db210506b6a3a8b0074f576ac369a8401d157aed4914d565180ccaf5e26f9 WatchSource:0}: Error finding container a86db210506b6a3a8b0074f576ac369a8401d157aed4914d565180ccaf5e26f9: Status 404 returned error can't find the container with id a86db210506b6a3a8b0074f576ac369a8401d157aed4914d565180ccaf5e26f9
Nov 03 10:36:13.658073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.658050    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30b7fb6a-bf01-4e56-b60e-dd62207d919c-webhook-cert\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.658174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.658102    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tksp7\" (UniqueName: \"kubernetes.io/projected/30b7fb6a-bf01-4e56-b60e-dd62207d919c-kube-api-access-tksp7\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.658213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.658201    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30b7fb6a-bf01-4e56-b60e-dd62207d919c-metrics-certs\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.759338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.759307    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30b7fb6a-bf01-4e56-b60e-dd62207d919c-metrics-certs\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.759512 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.759357    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30b7fb6a-bf01-4e56-b60e-dd62207d919c-webhook-cert\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.759512 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.759417    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tksp7\" (UniqueName: \"kubernetes.io/projected/30b7fb6a-bf01-4e56-b60e-dd62207d919c-kube-api-access-tksp7\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.762002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.761975    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30b7fb6a-bf01-4e56-b60e-dd62207d919c-metrics-certs\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.762002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.761996    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30b7fb6a-bf01-4e56-b60e-dd62207d919c-webhook-cert\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.767277 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.767254    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tksp7\" (UniqueName: \"kubernetes.io/projected/30b7fb6a-bf01-4e56-b60e-dd62207d919c-kube-api-access-tksp7\") pod \"maas-controller-6fdb79d597-9n4gw\" (UID: \"30b7fb6a-bf01-4e56-b60e-dd62207d919c\") " pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.853897 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.853864    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:13.910702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:13.910659    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67566d777-jjbdl" event={"ID":"f515dc20-2b5f-48cb-9623-8ea7440187e9","Type":"ContainerStarted","Data":"a86db210506b6a3a8b0074f576ac369a8401d157aed4914d565180ccaf5e26f9"}
Nov 03 10:36:14.127910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:14.127881    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fdb79d597-9n4gw"]
Nov 03 10:36:14.130298 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:36:14.130273    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b7fb6a_bf01_4e56_b60e_dd62207d919c.slice/crio-e366dabd7e229255ba4edee9f38787fc4e27f624eb9ae2df1897e0d0924ad3cf WatchSource:0}: Error finding container e366dabd7e229255ba4edee9f38787fc4e27f624eb9ae2df1897e0d0924ad3cf: Status 404 returned error can't find the container with id e366dabd7e229255ba4edee9f38787fc4e27f624eb9ae2df1897e0d0924ad3cf
Nov 03 10:36:14.917130 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:14.917093    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fdb79d597-9n4gw" event={"ID":"30b7fb6a-bf01-4e56-b60e-dd62207d919c","Type":"ContainerStarted","Data":"e366dabd7e229255ba4edee9f38787fc4e27f624eb9ae2df1897e0d0924ad3cf"}
Nov 03 10:36:17.554558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:17.554535    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 10:36:17.932145 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:17.932110    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fdb79d597-9n4gw" event={"ID":"30b7fb6a-bf01-4e56-b60e-dd62207d919c","Type":"ContainerStarted","Data":"a12610b6a0b319b23fd908bb7cc5fd28ff964f855777d1dc5bd8a2199daba194"}
Nov 03 10:36:17.932385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:17.932223    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:17.933630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:17.933603    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67566d777-jjbdl" event={"ID":"f515dc20-2b5f-48cb-9623-8ea7440187e9","Type":"ContainerStarted","Data":"447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35"}
Nov 03 10:36:17.933754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:17.933660    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:17.970283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:17.970227    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6fdb79d597-9n4gw" podStartSLOduration=1.5509073660000001 podStartE2EDuration="4.970211609s" totalImagesPullingTime="3.419304243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:17.967625473 +0000 UTC m=+421.872479913" watchObservedRunningTime="2026-11-03 10:36:17.970211609 +0000 UTC m=+421.875066085"
Nov 03 10:36:18.002631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:18.002580    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67566d777-jjbdl" podStartSLOduration=1.097005406 podStartE2EDuration="5.002566432s" totalImagesPullingTime="3.905561026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:18.001427117 +0000 UTC m=+421.906281582" watchObservedRunningTime="2026-11-03 10:36:18.002566432 +0000 UTC m=+421.907420873"
Nov 03 10:36:28.940662 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:28.940630    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fdb79d597-9n4gw"
Nov 03 10:36:28.941254 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:28.941237    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:29.000819 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.000781    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67566d777-jjbdl"]
Nov 03 10:36:29.001000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.000981    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-67566d777-jjbdl" podUID="f515dc20-2b5f-48cb-9623-8ea7440187e9" containerName="manager" containerID="cri-o://447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35" gracePeriod=30
Nov 03 10:36:29.246146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.246115    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:29.404197 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.404160    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f515dc20-2b5f-48cb-9623-8ea7440187e9-kube-api-access-cvhv4\" (UniqueName: \"kubernetes.io/projected/f515dc20-2b5f-48cb-9623-8ea7440187e9-kube-api-access-cvhv4\") pod \"f515dc20-2b5f-48cb-9623-8ea7440187e9\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") "
Nov 03 10:36:29.404372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.404274    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-metrics-certs\") pod \"f515dc20-2b5f-48cb-9623-8ea7440187e9\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") "
Nov 03 10:36:29.404372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.404304    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-webhook-cert\") pod \"f515dc20-2b5f-48cb-9623-8ea7440187e9\" (UID: \"f515dc20-2b5f-48cb-9623-8ea7440187e9\") "
Nov 03 10:36:29.406685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.406646    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-webhook-cert" pod "f515dc20-2b5f-48cb-9623-8ea7440187e9" (UID: "f515dc20-2b5f-48cb-9623-8ea7440187e9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:36:29.406685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.406678    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-metrics-certs" pod "f515dc20-2b5f-48cb-9623-8ea7440187e9" (UID: "f515dc20-2b5f-48cb-9623-8ea7440187e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:36:29.406835 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.406778    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f515dc20-2b5f-48cb-9623-8ea7440187e9-kube-api-access-cvhv4" pod "f515dc20-2b5f-48cb-9623-8ea7440187e9" (UID: "f515dc20-2b5f-48cb-9623-8ea7440187e9"). InnerVolumeSpecName "kube-api-access-cvhv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:36:29.505572 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.505474    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvhv4\" (UniqueName: \"kubernetes.io/projected/f515dc20-2b5f-48cb-9623-8ea7440187e9-kube-api-access-cvhv4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:29.505572 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.505528    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:29.505572 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.505538    5393 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f515dc20-2b5f-48cb-9623-8ea7440187e9-webhook-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:36:29.983183 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.983148    5393 generic.go:356] "Generic (PLEG): container finished" podID="f515dc20-2b5f-48cb-9623-8ea7440187e9" containerID="447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35" exitCode=0
Nov 03 10:36:29.983560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.983199    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67566d777-jjbdl"
Nov 03 10:36:29.983560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.983212    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67566d777-jjbdl" event={"ID":"f515dc20-2b5f-48cb-9623-8ea7440187e9","Type":"ContainerDied","Data":"447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35"}
Nov 03 10:36:29.983560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.983253    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67566d777-jjbdl" event={"ID":"f515dc20-2b5f-48cb-9623-8ea7440187e9","Type":"ContainerDied","Data":"a86db210506b6a3a8b0074f576ac369a8401d157aed4914d565180ccaf5e26f9"}
Nov 03 10:36:29.983560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.983273    5393 scope.go:122] "RemoveContainer" containerID="447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35"
Nov 03 10:36:29.991729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.991711    5393 scope.go:122] "RemoveContainer" containerID="447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35"
Nov 03 10:36:29.992005 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:36:29.991983    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35\": container with ID starting with 447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35 not found: ID does not exist" containerID="447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35"
Nov 03 10:36:29.992048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:29.992014    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35"} err="failed to get container status \"447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35\": rpc error: code = NotFound desc = could not find container \"447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35\": container with ID starting with 447d324e81b2f7e3c077a81e069402f4ec86101143803ba6f733999bada72a35 not found: ID does not exist"
Nov 03 10:36:30.004165 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:30.004138    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67566d777-jjbdl"]
Nov 03 10:36:30.009004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:30.008983    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67566d777-jjbdl"]
Nov 03 10:36:30.227794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:30.227764    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f515dc20-2b5f-48cb-9623-8ea7440187e9" path="/var/lib/kubelet/pods/f515dc20-2b5f-48cb-9623-8ea7440187e9/volumes"
Nov 03 10:36:35.873944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:35.873901    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:36:35.876841 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:35.876824    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xmpx9"
Nov 03 10:36:43.187629 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:36:43.187587    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:36:50.208751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.208718    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"]
Nov 03 10:36:50.209261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.209119    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f515dc20-2b5f-48cb-9623-8ea7440187e9" containerName="manager"
Nov 03 10:36:50.209261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.209134    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f515dc20-2b5f-48cb-9623-8ea7440187e9" containerName="manager"
Nov 03 10:36:50.209261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.209213    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f515dc20-2b5f-48cb-9623-8ea7440187e9" containerName="manager"
Nov 03 10:36:50.215718 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.215699    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.226059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.226030    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 10:36:50.226227 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.226210    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l8bvs\""
Nov 03 10:36:50.226730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.226706    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 10:36:50.229799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.229778    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"]
Nov 03 10:36:50.283680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.283645    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-metrics-certs\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.283847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.283687    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93288b8b-3afe-453d-b2fc-d97f7f29a507-openshift-service-ca\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.283847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.283812    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggv5\" (UniqueName: \"kubernetes.io/projected/93288b8b-3afe-453d-b2fc-d97f7f29a507-kube-api-access-zggv5\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.283847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.283842    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-maas-api-tls\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.384301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.384267    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93288b8b-3afe-453d-b2fc-d97f7f29a507-openshift-service-ca\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.384468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.384317    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggv5\" (UniqueName: \"kubernetes.io/projected/93288b8b-3afe-453d-b2fc-d97f7f29a507-kube-api-access-zggv5\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.384468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.384336    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-maas-api-tls\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.384468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.384372    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-metrics-certs\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.385051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.385029    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93288b8b-3afe-453d-b2fc-d97f7f29a507-openshift-service-ca\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.386953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.386908    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-maas-api-tls\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.387113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.387095    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-metrics-certs\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.392047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.392027    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggv5\" (UniqueName: \"kubernetes.io/projected/93288b8b-3afe-453d-b2fc-d97f7f29a507-kube-api-access-zggv5\") pod \"maas-api-6b77c88858-r4wnx\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") " pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.529232 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.529148    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:50.803366 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:50.803338    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"]
Nov 03 10:36:50.806089 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:36:50.806051    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93288b8b_3afe_453d_b2fc_d97f7f29a507.slice/crio-4936a52531238ebcd52dca238c71f687be696e103a3f35fa2f404e8fa9d815a5 WatchSource:0}: Error finding container 4936a52531238ebcd52dca238c71f687be696e103a3f35fa2f404e8fa9d815a5: Status 404 returned error can't find the container with id 4936a52531238ebcd52dca238c71f687be696e103a3f35fa2f404e8fa9d815a5
Nov 03 10:36:51.073710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:51.073631    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx" event={"ID":"93288b8b-3afe-453d-b2fc-d97f7f29a507","Type":"ContainerStarted","Data":"4936a52531238ebcd52dca238c71f687be696e103a3f35fa2f404e8fa9d815a5"}
Nov 03 10:36:55.092365 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:55.092323    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx" event={"ID":"93288b8b-3afe-453d-b2fc-d97f7f29a507","Type":"ContainerStarted","Data":"532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5"}
Nov 03 10:36:55.092802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:55.092413    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:36:55.109511 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:36:55.109464    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx" podStartSLOduration=1.3993033000000001 podStartE2EDuration="5.109450062s" totalImagesPullingTime="3.710146762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:36:55.107835911 +0000 UTC m=+459.012690363" watchObservedRunningTime="2026-11-03 10:36:55.109450062 +0000 UTC m=+459.014304503"
Nov 03 10:37:01.102436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:01.102407    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:37:02.025062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.025026    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"]
Nov 03 10:37:02.028316 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.028297    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.036680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.036657    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"]
Nov 03 10:37:02.082827 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.082792    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff27b47-b606-427c-ad74-87c4d5d0bd43-maas-api-tls\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.083059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.082838    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqktf\" (UniqueName: \"kubernetes.io/projected/7ff27b47-b606-427c-ad74-87c4d5d0bd43-kube-api-access-bqktf\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.083059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.082997    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff27b47-b606-427c-ad74-87c4d5d0bd43-metrics-certs\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.083157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.083082    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff27b47-b606-427c-ad74-87c4d5d0bd43-openshift-service-ca\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.184042 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.184003    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff27b47-b606-427c-ad74-87c4d5d0bd43-maas-api-tls\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.184505 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.184057    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqktf\" (UniqueName: \"kubernetes.io/projected/7ff27b47-b606-427c-ad74-87c4d5d0bd43-kube-api-access-bqktf\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.184505 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.184084    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff27b47-b606-427c-ad74-87c4d5d0bd43-metrics-certs\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.184505 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.184113    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff27b47-b606-427c-ad74-87c4d5d0bd43-openshift-service-ca\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.184768 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.184747    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff27b47-b606-427c-ad74-87c4d5d0bd43-openshift-service-ca\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.186625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.186604    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff27b47-b606-427c-ad74-87c4d5d0bd43-maas-api-tls\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.186625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.186621    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff27b47-b606-427c-ad74-87c4d5d0bd43-metrics-certs\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.194360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.194330    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqktf\" (UniqueName: \"kubernetes.io/projected/7ff27b47-b606-427c-ad74-87c4d5d0bd43-kube-api-access-bqktf\") pod \"maas-api-55d5b7b94d-v7gmn\" (UID: \"7ff27b47-b606-427c-ad74-87c4d5d0bd43\") " pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.342080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.341991    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:02.613080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:02.612964    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"]
Nov 03 10:37:02.615745 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:37:02.615723    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff27b47_b606_427c_ad74_87c4d5d0bd43.slice/crio-ab413415cd85daf06212b76d47929c048af1fcb487939cee55985f24799fa56a WatchSource:0}: Error finding container ab413415cd85daf06212b76d47929c048af1fcb487939cee55985f24799fa56a: Status 404 returned error can't find the container with id ab413415cd85daf06212b76d47929c048af1fcb487939cee55985f24799fa56a
Nov 03 10:37:03.125080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:03.125045    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn" event={"ID":"7ff27b47-b606-427c-ad74-87c4d5d0bd43","Type":"ContainerStarted","Data":"fbf728f701ca5608c1d3c97be2b422100e9af08c02bd798fed963fb8cfabe55f"}
Nov 03 10:37:03.125249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:03.125105    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn" event={"ID":"7ff27b47-b606-427c-ad74-87c4d5d0bd43","Type":"ContainerStarted","Data":"ab413415cd85daf06212b76d47929c048af1fcb487939cee55985f24799fa56a"}
Nov 03 10:37:03.125249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:03.125143    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:03.142570 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:03.142512    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn" podStartSLOduration=0.724480555 podStartE2EDuration="1.142495016s" totalImagesPullingTime="418.014461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:37:03.142141571 +0000 UTC m=+467.046996023" watchObservedRunningTime="2026-11-03 10:37:03.142495016 +0000 UTC m=+467.047349457"
Nov 03 10:37:05.588329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.588301    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"]
Nov 03 10:37:05.592076 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.592056    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:05.599633 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.599533    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"]
Nov 03 10:37:05.614820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.614785    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tnr\" (UniqueName: \"kubernetes.io/projected/86002e1a-3c47-4d7c-ae60-461da50a612c-kube-api-access-b5tnr\") pod \"llmisvc-controller-manager-6466c554d8-rsn52\" (UID: \"86002e1a-3c47-4d7c-ae60-461da50a612c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:05.615004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.614904    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86002e1a-3c47-4d7c-ae60-461da50a612c-cert\") pod \"llmisvc-controller-manager-6466c554d8-rsn52\" (UID: \"86002e1a-3c47-4d7c-ae60-461da50a612c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:05.716338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.716303    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tnr\" (UniqueName: \"kubernetes.io/projected/86002e1a-3c47-4d7c-ae60-461da50a612c-kube-api-access-b5tnr\") pod \"llmisvc-controller-manager-6466c554d8-rsn52\" (UID: \"86002e1a-3c47-4d7c-ae60-461da50a612c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:05.716517 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.716352    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86002e1a-3c47-4d7c-ae60-461da50a612c-cert\") pod \"llmisvc-controller-manager-6466c554d8-rsn52\" (UID: \"86002e1a-3c47-4d7c-ae60-461da50a612c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:05.718957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.718936    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86002e1a-3c47-4d7c-ae60-461da50a612c-cert\") pod \"llmisvc-controller-manager-6466c554d8-rsn52\" (UID: \"86002e1a-3c47-4d7c-ae60-461da50a612c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:05.723866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.723845    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tnr\" (UniqueName: \"kubernetes.io/projected/86002e1a-3c47-4d7c-ae60-461da50a612c-kube-api-access-b5tnr\") pod \"llmisvc-controller-manager-6466c554d8-rsn52\" (UID: \"86002e1a-3c47-4d7c-ae60-461da50a612c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:05.906218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:05.906172    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:06.176506 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:06.176427    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"]
Nov 03 10:37:06.179285 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:37:06.179253    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86002e1a_3c47_4d7c_ae60_461da50a612c.slice/crio-dead33ca505a3e365bb44ce3b68e8ce6256ca6dae0cf1187fa60e5afde05b0f0 WatchSource:0}: Error finding container dead33ca505a3e365bb44ce3b68e8ce6256ca6dae0cf1187fa60e5afde05b0f0: Status 404 returned error can't find the container with id dead33ca505a3e365bb44ce3b68e8ce6256ca6dae0cf1187fa60e5afde05b0f0
Nov 03 10:37:07.144413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:07.144381    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52" event={"ID":"86002e1a-3c47-4d7c-ae60-461da50a612c","Type":"ContainerStarted","Data":"e264e7cf37abaeaabbab735a04909c558c2560142d7b7179be8e878d54194284"}
Nov 03 10:37:07.144792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:07.144430    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52" event={"ID":"86002e1a-3c47-4d7c-ae60-461da50a612c","Type":"ContainerStarted","Data":"dead33ca505a3e365bb44ce3b68e8ce6256ca6dae0cf1187fa60e5afde05b0f0"}
Nov 03 10:37:07.144792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:07.144470    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:07.161683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:07.161637    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52" podStartSLOduration=1.782032262 podStartE2EDuration="2.161622828s" totalImagesPullingTime="379.590566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:37:07.160060906 +0000 UTC m=+471.064915348" watchObservedRunningTime="2026-11-03 10:37:07.161622828 +0000 UTC m=+471.066477309"
Nov 03 10:37:09.134648 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.134577    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55d5b7b94d-v7gmn"
Nov 03 10:37:09.178858 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.178822    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"]
Nov 03 10:37:09.179117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.179088    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx" podUID="93288b8b-3afe-453d-b2fc-d97f7f29a507" containerName="maas-api" containerID="cri-o://532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5" gracePeriod=30
Nov 03 10:37:09.423603 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.423576    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:37:09.453418 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.453388    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93288b8b-3afe-453d-b2fc-d97f7f29a507-kube-api-access-zggv5\" (UniqueName: \"kubernetes.io/projected/93288b8b-3afe-453d-b2fc-d97f7f29a507-kube-api-access-zggv5\") pod \"93288b8b-3afe-453d-b2fc-d97f7f29a507\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") "
Nov 03 10:37:09.453567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.453481    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-metrics-certs\") pod \"93288b8b-3afe-453d-b2fc-d97f7f29a507\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") "
Nov 03 10:37:09.453567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.453535    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93288b8b-3afe-453d-b2fc-d97f7f29a507-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93288b8b-3afe-453d-b2fc-d97f7f29a507-openshift-service-ca\") pod \"93288b8b-3afe-453d-b2fc-d97f7f29a507\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") "
Nov 03 10:37:09.453652 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.453571    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-maas-api-tls\") pod \"93288b8b-3afe-453d-b2fc-d97f7f29a507\" (UID: \"93288b8b-3afe-453d-b2fc-d97f7f29a507\") "
Nov 03 10:37:09.453981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.453950    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93288b8b-3afe-453d-b2fc-d97f7f29a507-openshift-service-ca" pod "93288b8b-3afe-453d-b2fc-d97f7f29a507" (UID: "93288b8b-3afe-453d-b2fc-d97f7f29a507"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:37:09.455741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.455718    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93288b8b-3afe-453d-b2fc-d97f7f29a507-kube-api-access-zggv5" pod "93288b8b-3afe-453d-b2fc-d97f7f29a507" (UID: "93288b8b-3afe-453d-b2fc-d97f7f29a507"). InnerVolumeSpecName "kube-api-access-zggv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:37:09.455832 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.455756    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-maas-api-tls" pod "93288b8b-3afe-453d-b2fc-d97f7f29a507" (UID: "93288b8b-3afe-453d-b2fc-d97f7f29a507"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:37:09.455832 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.455787    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-metrics-certs" pod "93288b8b-3afe-453d-b2fc-d97f7f29a507" (UID: "93288b8b-3afe-453d-b2fc-d97f7f29a507"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:37:09.555314 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.555276    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:37:09.555314 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.555309    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93288b8b-3afe-453d-b2fc-d97f7f29a507-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:37:09.555314 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.555319    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93288b8b-3afe-453d-b2fc-d97f7f29a507-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:37:09.555538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:09.555328    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zggv5\" (UniqueName: \"kubernetes.io/projected/93288b8b-3afe-453d-b2fc-d97f7f29a507-kube-api-access-zggv5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:37:10.160358 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.160324    5393 generic.go:356] "Generic (PLEG): container finished" podID="93288b8b-3afe-453d-b2fc-d97f7f29a507" containerID="532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5" exitCode=0
Nov 03 10:37:10.160762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.160375    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"
Nov 03 10:37:10.160762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.160391    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx" event={"ID":"93288b8b-3afe-453d-b2fc-d97f7f29a507","Type":"ContainerDied","Data":"532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5"}
Nov 03 10:37:10.160762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.160436    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx" event={"ID":"93288b8b-3afe-453d-b2fc-d97f7f29a507","Type":"ContainerDied","Data":"4936a52531238ebcd52dca238c71f687be696e103a3f35fa2f404e8fa9d815a5"}
Nov 03 10:37:10.160762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.160457    5393 scope.go:122] "RemoveContainer" containerID="532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5"
Nov 03 10:37:10.169623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.169607    5393 scope.go:122] "RemoveContainer" containerID="532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5"
Nov 03 10:37:10.169908 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:37:10.169891    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5\": container with ID starting with 532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5 not found: ID does not exist" containerID="532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5"
Nov 03 10:37:10.169982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.169946    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5"} err="failed to get container status \"532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5\": rpc error: code = NotFound desc = could not find container \"532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5\": container with ID starting with 532a57c1b0aa19195c55d26f7055bd1e0be07d191aec7d5f96ed3b576b7e98d5 not found: ID does not exist"
Nov 03 10:37:10.181534 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.181484    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"]
Nov 03 10:37:10.182882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.182859    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b77c88858-r4wnx"]
Nov 03 10:37:10.228826 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:10.228797    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93288b8b-3afe-453d-b2fc-d97f7f29a507" path="/var/lib/kubelet/pods/93288b8b-3afe-453d-b2fc-d97f7f29a507/volumes"
Nov 03 10:37:13.187888 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:37:13.187853    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:37:17.192878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:17.192844    5393 generic.go:356] "Generic (PLEG): container finished" podID="86002e1a-3c47-4d7c-ae60-461da50a612c" containerID="e264e7cf37abaeaabbab735a04909c558c2560142d7b7179be8e878d54194284" exitCode=1
Nov 03 10:37:17.193274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:17.192932    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52" event={"ID":"86002e1a-3c47-4d7c-ae60-461da50a612c","Type":"ContainerDied","Data":"e264e7cf37abaeaabbab735a04909c558c2560142d7b7179be8e878d54194284"}
Nov 03 10:37:17.193321 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:17.193277    5393 scope.go:122] "RemoveContainer" containerID="e264e7cf37abaeaabbab735a04909c558c2560142d7b7179be8e878d54194284"
Nov 03 10:37:18.202635 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:18.202603    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52" event={"ID":"86002e1a-3c47-4d7c-ae60-461da50a612c","Type":"ContainerStarted","Data":"6ea8300e32b0d0fd268a5f13911b9fda8fedd8206bc0a2d1fda2afb0a7f43b43"}
Nov 03 10:37:18.203078 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:18.202819    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:43.187729 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:37:43.187685    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:37:49.210088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.210051    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsn52"
Nov 03 10:37:49.250173 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.250143    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"]
Nov 03 10:37:49.250403 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.250380    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs" podUID="7943ffbe-ca5b-4751-a47e-9a06b04473ea" containerName="manager" containerID="cri-o://de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56" gracePeriod=30
Nov 03 10:37:49.500138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.500111    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:37:49.591555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.591516    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7943ffbe-ca5b-4751-a47e-9a06b04473ea-kube-api-access-fh2vd\" (UniqueName: \"kubernetes.io/projected/7943ffbe-ca5b-4751-a47e-9a06b04473ea-kube-api-access-fh2vd\") pod \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\" (UID: \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\") "
Nov 03 10:37:49.591722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.591579    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7943ffbe-ca5b-4751-a47e-9a06b04473ea-cert\" (UniqueName: \"kubernetes.io/secret/7943ffbe-ca5b-4751-a47e-9a06b04473ea-cert\") pod \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\" (UID: \"7943ffbe-ca5b-4751-a47e-9a06b04473ea\") "
Nov 03 10:37:49.593659 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.593629    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7943ffbe-ca5b-4751-a47e-9a06b04473ea-kube-api-access-fh2vd" pod "7943ffbe-ca5b-4751-a47e-9a06b04473ea" (UID: "7943ffbe-ca5b-4751-a47e-9a06b04473ea"). InnerVolumeSpecName "kube-api-access-fh2vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:37:49.593791 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.593702    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7943ffbe-ca5b-4751-a47e-9a06b04473ea-cert" pod "7943ffbe-ca5b-4751-a47e-9a06b04473ea" (UID: "7943ffbe-ca5b-4751-a47e-9a06b04473ea"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:37:49.692354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.692306    5393 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7943ffbe-ca5b-4751-a47e-9a06b04473ea-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:37:49.692354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:49.692341    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh2vd\" (UniqueName: \"kubernetes.io/projected/7943ffbe-ca5b-4751-a47e-9a06b04473ea-kube-api-access-fh2vd\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:37:50.343832 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.343795    5393 generic.go:356] "Generic (PLEG): container finished" podID="7943ffbe-ca5b-4751-a47e-9a06b04473ea" containerID="de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56" exitCode=0
Nov 03 10:37:50.344351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.343841    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"
Nov 03 10:37:50.344351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.343854    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs" event={"ID":"7943ffbe-ca5b-4751-a47e-9a06b04473ea","Type":"ContainerDied","Data":"de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56"}
Nov 03 10:37:50.344351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.343908    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs" event={"ID":"7943ffbe-ca5b-4751-a47e-9a06b04473ea","Type":"ContainerDied","Data":"f29482fc378881fd7836a675d52c53e59bb6f6eb1d2166a2125754b0d3866941"}
Nov 03 10:37:50.344351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.343947    5393 scope.go:122] "RemoveContainer" containerID="de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56"
Nov 03 10:37:50.353072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.353053    5393 scope.go:122] "RemoveContainer" containerID="de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56"
Nov 03 10:37:50.353412 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:37:50.353390    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56\": container with ID starting with de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56 not found: ID does not exist" containerID="de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56"
Nov 03 10:37:50.353469 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.353422    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56"} err="failed to get container status \"de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56\": rpc error: code = NotFound desc = could not find container \"de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56\": container with ID starting with de783e194dec1945b17ddd0ecbcc781257eb34df3b7e0b2902d6011f0b891c56 not found: ID does not exist"
Nov 03 10:37:50.359266 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.359243    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"]
Nov 03 10:37:50.363276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:50.363254    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-b7xzs"]
Nov 03 10:37:52.228438 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:37:52.228406    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7943ffbe-ca5b-4751-a47e-9a06b04473ea" path="/var/lib/kubelet/pods/7943ffbe-ca5b-4751-a47e-9a06b04473ea/volumes"
Nov 03 10:38:13.188584 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:38:13.188540    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:38:43.189576 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:38:43.189535    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:38:54.202278 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.202244    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"]
Nov 03 10:38:54.202795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.202776    5393 cpu_manager.go:414] "containerMap: removing container" podUID="93288b8b-3afe-453d-b2fc-d97f7f29a507" containerName="maas-api"
Nov 03 10:38:54.202890 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.202796    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93288b8b-3afe-453d-b2fc-d97f7f29a507" containerName="maas-api"
Nov 03 10:38:54.202890 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.202819    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7943ffbe-ca5b-4751-a47e-9a06b04473ea" containerName="manager"
Nov 03 10:38:54.202890 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.202826    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7943ffbe-ca5b-4751-a47e-9a06b04473ea" containerName="manager"
Nov 03 10:38:54.203072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.202951    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="93288b8b-3afe-453d-b2fc-d97f7f29a507" containerName="maas-api"
Nov 03 10:38:54.203072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.202964    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7943ffbe-ca5b-4751-a47e-9a06b04473ea" containerName="manager"
Nov 03 10:38:54.206272 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.206255    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.218343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.218317    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"]
Nov 03 10:38:54.276857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.276825    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.277041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.276867    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.277041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.276896    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.277041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.277025    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.277155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.277094    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.277155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.277135    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.277221 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.277164    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.277221 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.277196    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjkp\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-kube-api-access-cmjkp\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.277287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.277265    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.300676 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.300651    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rhbxr\""
Nov 03 10:38:54.378293 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378257    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378293 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378299    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378317    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378507    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378544    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378574    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378600    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378626    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjkp\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-kube-api-access-cmjkp\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378876 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378739    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378876 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378804    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.378876 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378866    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.379051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.378887    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.379268 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.379081    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.379327 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.379293    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.380791 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.380766    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.381213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.381196    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.390411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.390382    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.390773 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.390750    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjkp\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-kube-api-access-cmjkp\") pod \"maas-default-gateway-openshift-default-6cc458d887-d7gsx\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.519808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.519729    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:54.796767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:54.796738    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"]
Nov 03 10:38:54.799207 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:38:54.799175    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ef1d3a_2ef8_40c8_818e_b35ecc26150b.slice/crio-ee1621285be07a73a2dc0cc4354b5b739cfadde1e0a166bff2c8ece9550acf24 WatchSource:0}: Error finding container ee1621285be07a73a2dc0cc4354b5b739cfadde1e0a166bff2c8ece9550acf24: Status 404 returned error can't find the container with id ee1621285be07a73a2dc0cc4354b5b739cfadde1e0a166bff2c8ece9550acf24
Nov 03 10:38:55.618467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:55.618431    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" event={"ID":"44ef1d3a-2ef8-40c8-818e-b35ecc26150b","Type":"ContainerStarted","Data":"ee1621285be07a73a2dc0cc4354b5b739cfadde1e0a166bff2c8ece9550acf24"}
Nov 03 10:38:57.857490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:57.857460    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:38:57.857747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:57.857528    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:38:57.857747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:57.857557    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:38:58.634173 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:58.634144    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" event={"ID":"44ef1d3a-2ef8-40c8-818e-b35ecc26150b","Type":"ContainerStarted","Data":"583800af586b3b77b1d3f776b03ad864b7482a41194ce006cabb13055211c103"}
Nov 03 10:38:58.654459 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:58.654412    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podStartSLOduration=1.598769208 podStartE2EDuration="4.654398248s" totalImagesPullingTime="3.05562904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:38:58.653640999 +0000 UTC m=+582.558495441" watchObservedRunningTime="2026-11-03 10:38:58.654398248 +0000 UTC m=+582.559252689"
Nov 03 10:38:59.520425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:59.520387    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:38:59.522030 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:59.522000    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d7gsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 10:38:59.522154 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:38:59.522060    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 10:39:00.520390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:00.520352    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d7gsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 10:39:00.520563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:00.520407    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 10:39:01.520404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:01.520368    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d7gsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 10:39:01.520579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:01.520423    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 10:39:02.521096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:02.521053    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d7gsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 10:39:02.521504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:02.521112    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 10:39:03.520864 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:03.520828    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d7gsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 10:39:03.521072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:03.520882    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 10:39:04.520615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:04.520576    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d7gsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 10:39:04.521152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:04.520637    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 10:39:06.083403 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:06.083367    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:39:06.083973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:06.083934    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:39:06.084739 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:06.084721    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 10:39:09.209963 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.209878    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"]
Nov 03 10:39:09.213355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.213334    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.224087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.224057    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"]
Nov 03 10:39:09.321356 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321319    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5170bdd1-3790-4d0a-867a-5e23959411bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.321356 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321354    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.321586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321375    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.321586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321392    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.321586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321519    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.321586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321538    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.321586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321556    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.321586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321575    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6bh\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-kube-api-access-hg6bh\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.321859 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.321592    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.422699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.422652    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.422699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.422703    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.422836    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.422889    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6bh\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-kube-api-access-hg6bh\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.422942    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423061    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5170bdd1-3790-4d0a-867a-5e23959411bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423090    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423124    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423149    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423227    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423440 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423424    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423476 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423442    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423574 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423546    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.423720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.423704    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5170bdd1-3790-4d0a-867a-5e23959411bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.425351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.425325    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.425623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.425601    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.430542 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.430517    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.430706 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.430687    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6bh\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-kube-api-access-hg6bh\") pod \"maas-default-gateway-openshift-default-6cc458d887-tq59l\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.531738 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.531650    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:09.807120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.807081    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"]
Nov 03 10:39:09.809409 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:39:09.809375    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5170bdd1_3790_4d0a_867a_5e23959411bd.slice/crio-c0368b68427f7001e124831d18adef29a7f9d3dff73808d8f802c75f08616c0b WatchSource:0}: Error finding container c0368b68427f7001e124831d18adef29a7f9d3dff73808d8f802c75f08616c0b: Status 404 returned error can't find the container with id c0368b68427f7001e124831d18adef29a7f9d3dff73808d8f802c75f08616c0b
Nov 03 10:39:09.812627 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.812593    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:09.812722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.812673    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:09.812760 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:09.812738    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:10.709521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:10.709487    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" event={"ID":"5170bdd1-3790-4d0a-867a-5e23959411bd","Type":"ContainerStarted","Data":"9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28"}
Nov 03 10:39:10.709983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:10.709548    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" event={"ID":"5170bdd1-3790-4d0a-867a-5e23959411bd","Type":"ContainerStarted","Data":"c0368b68427f7001e124831d18adef29a7f9d3dff73808d8f802c75f08616c0b"}
Nov 03 10:39:10.730122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:10.730068    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podStartSLOduration=1.7300531559999999 podStartE2EDuration="1.730053156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:39:10.728724401 +0000 UTC m=+594.633578843" watchObservedRunningTime="2026-11-03 10:39:10.730053156 +0000 UTC m=+594.634907596"
Nov 03 10:39:11.532073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:11.532038    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:11.533538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:11.533516    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tq59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 10:39:11.533658 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:11.533562    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 10:39:12.532327 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:12.532289    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tq59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 10:39:12.532832 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:12.532350    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 10:39:13.211294 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:39:13.211249    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:39:13.532351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:13.532269    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tq59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 10:39:13.532351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:13.532321    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 10:39:14.532635 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:14.532599    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tq59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 10:39:14.533068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:14.532652    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 10:39:15.532796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:15.532758    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tq59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 10:39:15.533231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:15.532812    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 10:39:16.296500 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:16.296463    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:39:16.299828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:16.299802    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:39:16.532373 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:16.532334    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tq59l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 10:39:16.532565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:16.532400    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 10:39:17.997081 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:17.997046    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:17.997594 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:17.997392    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:17.998297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:17.998277    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 10:39:24.222954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.222899    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"]
Nov 03 10:39:24.226712 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.226690    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.236245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.236212    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"]
Nov 03 10:39:24.391292 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391249    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgwt9\" (UniqueName: \"kubernetes.io/projected/55911ac4-1644-448a-bf1e-056207cdb3b9-kube-api-access-bgwt9\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.391484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391302    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.391484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391387    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.391484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391423    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.391657 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391524    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.391657 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391620    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.391657 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391646    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.391779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391670    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55911ac4-1644-448a-bf1e-056207cdb3b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.391779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.391751    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.492863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.492765    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgwt9\" (UniqueName: \"kubernetes.io/projected/55911ac4-1644-448a-bf1e-056207cdb3b9-kube-api-access-bgwt9\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.492863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.492823    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.492863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.492849    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.492872    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.492904    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493017    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493039    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493060    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55911ac4-1644-448a-bf1e-056207cdb3b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493093    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493440    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493486    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493524    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493532    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.493909 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.493886    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55911ac4-1644-448a-bf1e-056207cdb3b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.495343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.495321    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.495626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.495607    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.501066 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.501044    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgwt9\" (UniqueName: \"kubernetes.io/projected/55911ac4-1644-448a-bf1e-056207cdb3b9-kube-api-access-bgwt9\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.501152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.501090    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55911ac4-1644-448a-bf1e-056207cdb3b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-74hrv\" (UID: \"55911ac4-1644-448a-bf1e-056207cdb3b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.541164 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.541138    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:24.815905 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.815883    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"]
Nov 03 10:39:24.818387 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:39:24.818357    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55911ac4_1644_448a_bf1e_056207cdb3b9.slice/crio-5919c8f80a9acb9c1e4504545c34cc65d7bb0a714ad3b2f2849e0498e09a6905 WatchSource:0}: Error finding container 5919c8f80a9acb9c1e4504545c34cc65d7bb0a714ad3b2f2849e0498e09a6905: Status 404 returned error can't find the container with id 5919c8f80a9acb9c1e4504545c34cc65d7bb0a714ad3b2f2849e0498e09a6905
Nov 03 10:39:24.820611 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.820580    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:24.820687 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.820642    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:24.820687 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:24.820670    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:39:25.777884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:25.777850    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" event={"ID":"55911ac4-1644-448a-bf1e-056207cdb3b9","Type":"ContainerStarted","Data":"9f12e1c9c8b66e50ce97e8c124b65dcc1ce33287e5c1631d8deee62b8284f875"}
Nov 03 10:39:25.778343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:25.777902    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" event={"ID":"55911ac4-1644-448a-bf1e-056207cdb3b9","Type":"ContainerStarted","Data":"5919c8f80a9acb9c1e4504545c34cc65d7bb0a714ad3b2f2849e0498e09a6905"}
Nov 03 10:39:25.818135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:25.818086    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" podStartSLOduration=1.818071902 podStartE2EDuration="1.818071902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:39:25.816014889 +0000 UTC m=+609.720869330" watchObservedRunningTime="2026-11-03 10:39:25.818071902 +0000 UTC m=+609.722926343"
Nov 03 10:39:26.541493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:26.541462    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:26.542619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:26.542594    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-74hrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 10:39:26.542739 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:26.542640    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" podUID="55911ac4-1644-448a-bf1e-056207cdb3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 10:39:27.542141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:27.542103    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-74hrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 10:39:27.542539 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:27.542159    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" podUID="55911ac4-1644-448a-bf1e-056207cdb3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 10:39:28.541685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:28.541649    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-74hrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 10:39:28.541896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:28.541715    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" podUID="55911ac4-1644-448a-bf1e-056207cdb3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 10:39:29.542202 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:29.542162    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-74hrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 10:39:29.542616 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:29.542214    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" podUID="55911ac4-1644-448a-bf1e-056207cdb3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 10:39:30.541711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:30.541671    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-74hrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 10:39:30.541895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:30.541759    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" podUID="55911ac4-1644-448a-bf1e-056207cdb3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 10:39:31.542049 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:31.542012    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-74hrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 10:39:31.542443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:31.542081    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" podUID="55911ac4-1644-448a-bf1e-056207cdb3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 10:39:32.987954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:32.987904    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:32.988402 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:32.988302    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:32.989204 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:39:32.989185    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv"
Nov 03 10:39:43.188601 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:39:43.188567    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:40:12.862938 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:12.862885    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"]
Nov 03 10:40:12.867244 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:12.867219    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:12.881126 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:12.881097    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"]
Nov 03 10:40:12.910655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:12.910626    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fc81ef02-serving-cert\""
Nov 03 10:40:12.949165 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:12.949130    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-maas-api-tls\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:12.949165 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:12.949163    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpz7b\" (UniqueName: \"kubernetes.io/projected/d18543b4-5dd1-4bdb-932a-f54b61f0e802-kube-api-access-vpz7b\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:12.949385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:12.949196    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-metrics-certs\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:12.949385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:12.949366    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d18543b4-5dd1-4bdb-932a-f54b61f0e802-openshift-service-ca\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.050465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.050431    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-maas-api-tls\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.050631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.050476    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpz7b\" (UniqueName: \"kubernetes.io/projected/d18543b4-5dd1-4bdb-932a-f54b61f0e802-kube-api-access-vpz7b\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.050631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.050514    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-metrics-certs\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.050631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.050624    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d18543b4-5dd1-4bdb-932a-f54b61f0e802-openshift-service-ca\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.051439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.051413    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d18543b4-5dd1-4bdb-932a-f54b61f0e802-openshift-service-ca\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.053397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.053371    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-metrics-certs\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.053523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.053498    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-maas-api-tls\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.058372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.058349    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpz7b\" (UniqueName: \"kubernetes.io/projected/d18543b4-5dd1-4bdb-932a-f54b61f0e802-kube-api-access-vpz7b\") pod \"maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.183035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.182951    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:13.221429 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:40:13.221395    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:40:13.483710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.483613    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"]
Nov 03 10:40:13.488603 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:40:13.488571    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18543b4_5dd1_4bdb_932a_f54b61f0e802.slice/crio-60a132fbb819b43e16ae446f4e668565adf7418b31076ba2e7f658ca655838a9 WatchSource:0}: Error finding container 60a132fbb819b43e16ae446f4e668565adf7418b31076ba2e7f658ca655838a9: Status 404 returned error can't find the container with id 60a132fbb819b43e16ae446f4e668565adf7418b31076ba2e7f658ca655838a9
Nov 03 10:40:13.490111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:13.490090    5393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:40:14.047332 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:14.047280    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r" event={"ID":"d18543b4-5dd1-4bdb-932a-f54b61f0e802","Type":"ContainerStarted","Data":"eb6e7bb95c682d9138e00271590af174bdbcfab9562e6e7f9393e65454ef1703"}
Nov 03 10:40:14.057685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:14.047381    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r" event={"ID":"d18543b4-5dd1-4bdb-932a-f54b61f0e802","Type":"ContainerStarted","Data":"60a132fbb819b43e16ae446f4e668565adf7418b31076ba2e7f658ca655838a9"}
Nov 03 10:40:14.057685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:14.048362    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:14.074708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:14.074620    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r" podStartSLOduration=1.641975194 podStartE2EDuration="2.074598749s" totalImagesPullingTime="432.623555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:14.072065958 +0000 UTC m=+657.976920399" watchObservedRunningTime="2026-11-03 10:40:14.074598749 +0000 UTC m=+657.979453191"
Nov 03 10:40:21.078373 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:21.078336    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:25.550736 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.550703    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"]
Nov 03 10:40:25.554873 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.554844    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.561085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.560683    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:40:25.561085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.560723    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:40:25.563292 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.563271    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"]
Nov 03 10:40:25.588591 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.588561    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.588781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.588648    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz42r\" (UniqueName: \"kubernetes.io/projected/e1155cef-3844-416f-8eff-ace3553b28c8-kube-api-access-wz42r\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.588781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.588756    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.689753 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.689720    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz42r\" (UniqueName: \"kubernetes.io/projected/e1155cef-3844-416f-8eff-ace3553b28c8-kube-api-access-wz42r\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.689952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.689765    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.689952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.689804    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.690601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.690530    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.692354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.692332    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.697287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.697268    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz42r\" (UniqueName: \"kubernetes.io/projected/e1155cef-3844-416f-8eff-ace3553b28c8-kube-api-access-wz42r\") pod \"maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:25.870520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:25.870483    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:26.143956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:26.143930    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"]
Nov 03 10:40:26.146407 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:40:26.146377    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1155cef_3844_416f_8eff_ace3553b28c8.slice/crio-170dd34e49a556889d8c3dfac62deb07dcd0330abe7beb7d258a34dc74c90539 WatchSource:0}: Error finding container 170dd34e49a556889d8c3dfac62deb07dcd0330abe7beb7d258a34dc74c90539: Status 404 returned error can't find the container with id 170dd34e49a556889d8c3dfac62deb07dcd0330abe7beb7d258a34dc74c90539
Nov 03 10:40:26.166282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:26.166248    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c" event={"ID":"e1155cef-3844-416f-8eff-ace3553b28c8","Type":"ContainerStarted","Data":"170dd34e49a556889d8c3dfac62deb07dcd0330abe7beb7d258a34dc74c90539"}
Nov 03 10:40:29.181640 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:29.181599    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c" event={"ID":"e1155cef-3844-416f-8eff-ace3553b28c8","Type":"ContainerStarted","Data":"e07e91b94fb8b4a207fc7eea671ef4de1c31563d3ed008b2a6692f8e5bcb840f"}
Nov 03 10:40:29.198662 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:29.198605    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c" podStartSLOduration=1.6599366340000001 podStartE2EDuration="4.198589006s" totalImagesPullingTime="2.538652372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:29.197542846 +0000 UTC m=+673.102397289" watchObservedRunningTime="2026-11-03 10:40:29.198589006 +0000 UTC m=+673.103443448"
Nov 03 10:40:30.188195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:30.188157    5393 generic.go:356] "Generic (PLEG): container finished" podID="e1155cef-3844-416f-8eff-ace3553b28c8" containerID="e07e91b94fb8b4a207fc7eea671ef4de1c31563d3ed008b2a6692f8e5bcb840f" exitCode=0
Nov 03 10:40:30.188573 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:30.188232    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c" event={"ID":"e1155cef-3844-416f-8eff-ace3553b28c8","Type":"ContainerDied","Data":"e07e91b94fb8b4a207fc7eea671ef4de1c31563d3ed008b2a6692f8e5bcb840f"}
Nov 03 10:40:31.349377 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.349345    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:31.449642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.449553    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1155cef-3844-416f-8eff-ace3553b28c8-kube-api-access-wz42r\" (UniqueName: \"kubernetes.io/projected/e1155cef-3844-416f-8eff-ace3553b28c8-kube-api-access-wz42r\") pod \"e1155cef-3844-416f-8eff-ace3553b28c8\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") "
Nov 03 10:40:31.449807 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.449646    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-xks-ca\") pod \"e1155cef-3844-416f-8eff-ace3553b28c8\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") "
Nov 03 10:40:31.449807 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.449685    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-openshift-ca\") pod \"e1155cef-3844-416f-8eff-ace3553b28c8\" (UID: \"e1155cef-3844-416f-8eff-ace3553b28c8\") "
Nov 03 10:40:31.450138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.450104    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-openshift-ca" pod "e1155cef-3844-416f-8eff-ace3553b28c8" (UID: "e1155cef-3844-416f-8eff-ace3553b28c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:31.451951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.451905    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-xks-ca" pod "e1155cef-3844-416f-8eff-ace3553b28c8" (UID: "e1155cef-3844-416f-8eff-ace3553b28c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:31.452061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.452024    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1155cef-3844-416f-8eff-ace3553b28c8-kube-api-access-wz42r" pod "e1155cef-3844-416f-8eff-ace3553b28c8" (UID: "e1155cef-3844-416f-8eff-ace3553b28c8"). InnerVolumeSpecName "kube-api-access-wz42r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:31.551558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.551523    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz42r\" (UniqueName: \"kubernetes.io/projected/e1155cef-3844-416f-8eff-ace3553b28c8-kube-api-access-wz42r\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:31.551558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.551558    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:31.551750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:31.551573    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1155cef-3844-416f-8eff-ace3553b28c8-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:32.199149 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:32.199105    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c" event={"ID":"e1155cef-3844-416f-8eff-ace3553b28c8","Type":"ContainerDied","Data":"170dd34e49a556889d8c3dfac62deb07dcd0330abe7beb7d258a34dc74c90539"}
Nov 03 10:40:32.199333 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:32.199162    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170dd34e49a556889d8c3dfac62deb07dcd0330abe7beb7d258a34dc74c90539"
Nov 03 10:40:32.199333 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:32.199117    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"
Nov 03 10:40:40.014447 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.014408    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"]
Nov 03 10:40:40.015082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.014738    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r" podUID="d18543b4-5dd1-4bdb-932a-f54b61f0e802" containerName="maas-api" containerID="cri-o://eb6e7bb95c682d9138e00271590af174bdbcfab9562e6e7f9393e65454ef1703" gracePeriod=30
Nov 03 10:40:40.240219 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.240037    5393 generic.go:356] "Generic (PLEG): container finished" podID="d18543b4-5dd1-4bdb-932a-f54b61f0e802" containerID="eb6e7bb95c682d9138e00271590af174bdbcfab9562e6e7f9393e65454ef1703" exitCode=0
Nov 03 10:40:40.240219 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.240152    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r" event={"ID":"d18543b4-5dd1-4bdb-932a-f54b61f0e802","Type":"ContainerDied","Data":"eb6e7bb95c682d9138e00271590af174bdbcfab9562e6e7f9393e65454ef1703"}
Nov 03 10:40:40.293239 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.293202    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:40.417358 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.417316    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"]
Nov 03 10:40:40.417532 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.417381    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fc81ef02-2c99c"]
Nov 03 10:40:40.447055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.446221    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-metrics-certs\") pod \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") "
Nov 03 10:40:40.447055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.446324    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18543b4-5dd1-4bdb-932a-f54b61f0e802-kube-api-access-vpz7b\" (UniqueName: \"kubernetes.io/projected/d18543b4-5dd1-4bdb-932a-f54b61f0e802-kube-api-access-vpz7b\") pod \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") "
Nov 03 10:40:40.447055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.446384    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-maas-api-tls\") pod \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") "
Nov 03 10:40:40.447055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.446427    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18543b4-5dd1-4bdb-932a-f54b61f0e802-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d18543b4-5dd1-4bdb-932a-f54b61f0e802-openshift-service-ca\") pod \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\" (UID: \"d18543b4-5dd1-4bdb-932a-f54b61f0e802\") "
Nov 03 10:40:40.447911 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.447604    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18543b4-5dd1-4bdb-932a-f54b61f0e802-openshift-service-ca" pod "d18543b4-5dd1-4bdb-932a-f54b61f0e802" (UID: "d18543b4-5dd1-4bdb-932a-f54b61f0e802"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:40:40.459271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.450717    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18543b4-5dd1-4bdb-932a-f54b61f0e802-kube-api-access-vpz7b" pod "d18543b4-5dd1-4bdb-932a-f54b61f0e802" (UID: "d18543b4-5dd1-4bdb-932a-f54b61f0e802"). InnerVolumeSpecName "kube-api-access-vpz7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:40:40.461110 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.461049    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-maas-api-tls" pod "d18543b4-5dd1-4bdb-932a-f54b61f0e802" (UID: "d18543b4-5dd1-4bdb-932a-f54b61f0e802"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:40.461378 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.461358    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-metrics-certs" pod "d18543b4-5dd1-4bdb-932a-f54b61f0e802" (UID: "d18543b4-5dd1-4bdb-932a-f54b61f0e802"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:40:40.547567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.547486    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpz7b\" (UniqueName: \"kubernetes.io/projected/d18543b4-5dd1-4bdb-932a-f54b61f0e802-kube-api-access-vpz7b\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:40.547567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.547526    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:40.547567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.547544    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d18543b4-5dd1-4bdb-932a-f54b61f0e802-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:40.547567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:40.547558    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d18543b4-5dd1-4bdb-932a-f54b61f0e802-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:40:41.246338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:41.246302    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r" event={"ID":"d18543b4-5dd1-4bdb-932a-f54b61f0e802","Type":"ContainerDied","Data":"60a132fbb819b43e16ae446f4e668565adf7418b31076ba2e7f658ca655838a9"}
Nov 03 10:40:41.246338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:41.246338    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"
Nov 03 10:40:41.246843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:41.246361    5393 scope.go:122] "RemoveContainer" containerID="eb6e7bb95c682d9138e00271590af174bdbcfab9562e6e7f9393e65454ef1703"
Nov 03 10:40:41.273110 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:41.272676    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"]
Nov 03 10:40:41.274359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:41.274338    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fc81ef02-7679c747bf-rmw9r"]
Nov 03 10:40:42.231950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.231874    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18543b4-5dd1-4bdb-932a-f54b61f0e802" path="/var/lib/kubelet/pods/d18543b4-5dd1-4bdb-932a-f54b61f0e802/volumes"
Nov 03 10:40:42.232466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.232446    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1155cef-3844-416f-8eff-ace3553b28c8" path="/var/lib/kubelet/pods/e1155cef-3844-416f-8eff-ace3553b28c8/volumes"
Nov 03 10:40:42.252163 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.252132    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"]
Nov 03 10:40:42.252617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.252528    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e1155cef-3844-416f-8eff-ace3553b28c8" containerName="revoke-keys"
Nov 03 10:40:42.252617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.252542    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1155cef-3844-416f-8eff-ace3553b28c8" containerName="revoke-keys"
Nov 03 10:40:42.252617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.252560    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d18543b4-5dd1-4bdb-932a-f54b61f0e802" containerName="maas-api"
Nov 03 10:40:42.252617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.252565    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18543b4-5dd1-4bdb-932a-f54b61f0e802" containerName="maas-api"
Nov 03 10:40:42.252828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.252636    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1155cef-3844-416f-8eff-ace3553b28c8" containerName="revoke-keys"
Nov 03 10:40:42.252828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.252645    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18543b4-5dd1-4bdb-932a-f54b61f0e802" containerName="maas-api"
Nov 03 10:40:42.257125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.257101    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.271359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.271333    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"]
Nov 03 10:40:42.322891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.322855    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d5d3c3-openshift-default-dockercfg-wbpds\""
Nov 03 10:40:42.365718 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.365681    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.365905 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.365757    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.365905 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.365887    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.366044 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.365949    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.366100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.366072    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.366150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.366128    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh242\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-kube-api-access-vh242\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.366186 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.366165    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.366240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.366188    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.366291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.366249    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.468374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.468332    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.469101 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.469078    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.469286 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.469270    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.469429 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.469413    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh242\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-kube-api-access-vh242\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.469541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.469528    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.469675 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.469637    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.469786 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.469773    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.469939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.469902    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.470092 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.470075    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.470490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.470470    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.470611 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.469020    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.474622 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.471292    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.474787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.473220    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.474907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.474481    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.476653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.476626    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.477288 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.477264    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.483515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.483453    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.484092 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.484070    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh242\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-kube-api-access-vh242\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.575311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.575274    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:42.884064 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.884041    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"]
Nov 03 10:40:42.886900 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:40:42.886876    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648d3e6d_2f48_4a99_b293_9b124b51ebf1.slice/crio-0e17babdb8d22b07dd28770d391b5b2c85d4cbe9de6eb7fb63934406edc0052f WatchSource:0}: Error finding container 0e17babdb8d22b07dd28770d391b5b2c85d4cbe9de6eb7fb63934406edc0052f: Status 404 returned error can't find the container with id 0e17babdb8d22b07dd28770d391b5b2c85d4cbe9de6eb7fb63934406edc0052f
Nov 03 10:40:42.889505 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.889471    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:42.889609 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.889549    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:42.889609 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:42.889593    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:43.236484 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:40:43.236384    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:40:43.262863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:43.262825    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" event={"ID":"648d3e6d-2f48-4a99-b293-9b124b51ebf1","Type":"ContainerStarted","Data":"f3f4d76edef6865ea7eb4859dbc6c742b3355bcbd6ff9bb5f15530cc20c4d974"}
Nov 03 10:40:43.263274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:43.262881    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" event={"ID":"648d3e6d-2f48-4a99-b293-9b124b51ebf1","Type":"ContainerStarted","Data":"0e17babdb8d22b07dd28770d391b5b2c85d4cbe9de6eb7fb63934406edc0052f"}
Nov 03 10:40:43.286114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:43.286048    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podStartSLOduration=1.286027485 podStartE2EDuration="1.286027485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:43.28345786 +0000 UTC m=+687.188312326" watchObservedRunningTime="2026-11-03 10:40:43.286027485 +0000 UTC m=+687.190881939"
Nov 03 10:40:43.576556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:43.576378    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:43.604684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:43.600214    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:44.269003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.268969    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:44.271440 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.271214    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:40:44.275549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.273738    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"]
Nov 03 10:40:44.277956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.277910    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.296152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.294717    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f78621-openshift-default-dockercfg-2x24f\""
Nov 03 10:40:44.297212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.297189    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"]
Nov 03 10:40:44.398201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.398166    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.398380 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.398241    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.398544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.398372    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.398544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.398434    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.398544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.398457    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6469e418-320e-4d01-86a4-99b169b7fc34-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.398544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.398483    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnm9z\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-kube-api-access-bnm9z\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.398544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.398530    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6469e418-320e-4d01-86a4-99b169b7fc34-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.399284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.399088    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.399284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.399150    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.500664 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.500624    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.500664 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.500667    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.500964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.500710    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.500964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.500757    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.500964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.500789    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.500964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.500896    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.500964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.500958    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6469e418-320e-4d01-86a4-99b169b7fc34-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.501229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.501000    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnm9z\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-kube-api-access-bnm9z\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.501229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.501048    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6469e418-320e-4d01-86a4-99b169b7fc34-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.501330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.501242    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.501587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.501564    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.501742 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.501643    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.501800 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.501792    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.501995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.501972    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6469e418-320e-4d01-86a4-99b169b7fc34-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.503453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.503425    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.504011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.503990    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6469e418-320e-4d01-86a4-99b169b7fc34-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.509145 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.509122    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnm9z\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-kube-api-access-bnm9z\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.509649 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.509628    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.639951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.639344    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:44.927623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.927596    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"]
Nov 03 10:40:44.934718 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:40:44.934691    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6469e418_320e_4d01_86a4_99b169b7fc34.slice/crio-c6e2d9d0f3456ec8370294c90129d74329e213ca415d9d1fc1b0b610a349379c WatchSource:0}: Error finding container c6e2d9d0f3456ec8370294c90129d74329e213ca415d9d1fc1b0b610a349379c: Status 404 returned error can't find the container with id c6e2d9d0f3456ec8370294c90129d74329e213ca415d9d1fc1b0b610a349379c
Nov 03 10:40:44.937500 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.937468    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:44.937601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.937537    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:44.937601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:44.937566    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:40:45.275753 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:45.275649    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" event={"ID":"6469e418-320e-4d01-86a4-99b169b7fc34","Type":"ContainerStarted","Data":"ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296"}
Nov 03 10:40:45.275753 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:45.275709    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" event={"ID":"6469e418-320e-4d01-86a4-99b169b7fc34","Type":"ContainerStarted","Data":"c6e2d9d0f3456ec8370294c90129d74329e213ca415d9d1fc1b0b610a349379c"}
Nov 03 10:40:45.305014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:45.304947    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" podStartSLOduration=1.30491288 podStartE2EDuration="1.30491288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:40:45.303123384 +0000 UTC m=+689.207977825" watchObservedRunningTime="2026-11-03 10:40:45.30491288 +0000 UTC m=+689.209767321"
Nov 03 10:40:45.640234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:45.640201    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:46.646109 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:46.646081    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:47.287006 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:47.286975    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:47.287881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:47.287859    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:40:59.993196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:59.992907    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:41852->10.134.0.40:15021: read: connection reset by peer" start-of-body=
Nov 03 10:40:59.993196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:40:59.993005    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:41852->10.134.0.40:15021: read: connection reset by peer"
Nov 03 10:41:00.352788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:00.352757    5393 generic.go:356] "Generic (PLEG): container finished" podID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerID="f3f4d76edef6865ea7eb4859dbc6c742b3355bcbd6ff9bb5f15530cc20c4d974" exitCode=0
Nov 03 10:41:00.353014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:00.352836    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" event={"ID":"648d3e6d-2f48-4a99-b293-9b124b51ebf1","Type":"ContainerDied","Data":"f3f4d76edef6865ea7eb4859dbc6c742b3355bcbd6ff9bb5f15530cc20c4d974"}
Nov 03 10:41:00.353292 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:00.353263    5393 scope.go:122] "RemoveContainer" containerID="f3f4d76edef6865ea7eb4859dbc6c742b3355bcbd6ff9bb5f15530cc20c4d974"
Nov 03 10:41:00.354568 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:00.354541    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:00.354638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:00.354611    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:00.354677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:00.354645    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:01.361147 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.361106    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" event={"ID":"648d3e6d-2f48-4a99-b293-9b124b51ebf1","Type":"ContainerStarted","Data":"4cd6dd54a0f52d65b7b541c676b7d290fb138a306f1385163dc0f2dc8f426fc9"}
Nov 03 10:41:01.575548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.575512    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:41:01.575850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.575821    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 10:41:01.575976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.575874    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 10:41:01.727436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.727352    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"]
Nov 03 10:41:01.730583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.730559    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.743052 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.743022    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"]
Nov 03 10:41:01.743195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.743155    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"simulator-w1-010232-kserve-self-signed-certs\""
Nov 03 10:41:01.743256 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.743230    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"default-dockercfg-pfvjf\""
Nov 03 10:41:01.743303 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.743283    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"kube-root-ca.crt\""
Nov 03 10:41:01.743596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.743579    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"openshift-service-ca.crt\""
Nov 03 10:41:01.827565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.827509    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-tmp-dir\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.827751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.827622    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271be97a-c7bd-48fc-93c4-99e7a1de452a-tls-certs\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.827751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.827680    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-dshm\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.827751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.827707    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lvr\" (UniqueName: \"kubernetes.io/projected/271be97a-c7bd-48fc-93c4-99e7a1de452a-kube-api-access-c5lvr\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.827878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.827819    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-home\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.827878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.827852    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-model-cache\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.930610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.928835    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271be97a-c7bd-48fc-93c4-99e7a1de452a-tls-certs\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.930610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.928954    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-dshm\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.930610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.928995    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lvr\" (UniqueName: \"kubernetes.io/projected/271be97a-c7bd-48fc-93c4-99e7a1de452a-kube-api-access-c5lvr\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.930610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.929053    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-home\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.930610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.929075    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-model-cache\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.930610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.929157    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-tmp-dir\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.930610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.929579    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-tmp-dir\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.930610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.930541    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-model-cache\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.931150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.930674    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-home\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.932891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.932858    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271be97a-c7bd-48fc-93c4-99e7a1de452a-tls-certs\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.937675 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.933371    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-dshm\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:01.946955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:01.939686    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lvr\" (UniqueName: \"kubernetes.io/projected/271be97a-c7bd-48fc-93c4-99e7a1de452a-kube-api-access-c5lvr\") pod \"simulator-w1-010232-kserve-88cb857c6-pqtlk\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") " pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:02.044751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:02.044663    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:02.353299 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:02.353269    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"]
Nov 03 10:41:02.356631 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:41:02.356595    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod271be97a_c7bd_48fc_93c4_99e7a1de452a.slice/crio-66a4ff43495f55a4e1f3b3f533e9673c616eeb4ce8263ecef34f68d223903639 WatchSource:0}: Error finding container 66a4ff43495f55a4e1f3b3f533e9673c616eeb4ce8263ecef34f68d223903639: Status 404 returned error can't find the container with id 66a4ff43495f55a4e1f3b3f533e9673c616eeb4ce8263ecef34f68d223903639
Nov 03 10:41:02.368273 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:02.368245    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" event={"ID":"271be97a-c7bd-48fc-93c4-99e7a1de452a","Type":"ContainerStarted","Data":"66a4ff43495f55a4e1f3b3f533e9673c616eeb4ce8263ecef34f68d223903639"}
Nov 03 10:41:02.578447 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:02.578413    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 10:41:02.578822 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:02.578778    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 10:41:03.576487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:03.576441    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 10:41:03.578986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:03.576506    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 10:41:04.576079 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:04.576034    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 10:41:04.576251 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:04.576096    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 10:41:05.419045 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:05.419008    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" event={"ID":"271be97a-c7bd-48fc-93c4-99e7a1de452a","Type":"ContainerStarted","Data":"cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6"}
Nov 03 10:41:05.437369 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:05.437319    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podStartSLOduration=2.430236006 podStartE2EDuration="4.437294697s" totalImagesPullingTime="2.007058691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:05.435317347 +0000 UTC m=+709.340171788" watchObservedRunningTime="2026-11-03 10:41:05.437294697 +0000 UTC m=+709.342149135"
Nov 03 10:41:05.578756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:05.576297    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 10:41:05.578756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:05.576346    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 10:41:06.045813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:06.045766    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:06.601945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:06.600236    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:41:06.601945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:06.601594    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:41:06.609943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:06.606274    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 10:41:12.045951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:12.045839    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:12.060465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:12.060436    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:12.466494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:12.466464    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:41:13.192590 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:41:13.192548    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:41:15.471510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:15.471482    5393 generic.go:356] "Generic (PLEG): container finished" podID="6469e418-320e-4d01-86a4-99b169b7fc34" containerID="ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296" exitCode=0
Nov 03 10:41:15.472043 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:15.471558    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" event={"ID":"6469e418-320e-4d01-86a4-99b169b7fc34","Type":"ContainerDied","Data":"ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296"}
Nov 03 10:41:15.472043 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:15.471981    5393 scope.go:122] "RemoveContainer" containerID="ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296"
Nov 03 10:41:15.473206 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:15.473180    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:15.473287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:15.473242    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:15.473287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:15.473273    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:16.479001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:16.478966    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" event={"ID":"6469e418-320e-4d01-86a4-99b169b7fc34","Type":"ContainerStarted","Data":"2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216"}
Nov 03 10:41:16.640333 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:16.640294    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:41:16.641646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:16.641620    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:41:16.641760 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:16.641670    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 10:41:17.288118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:17.288027    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:41:17.640226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:17.640194    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:41:17.640623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:17.640246    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 10:41:18.640599 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:18.640558    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:41:18.641033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:18.640623    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 10:41:19.640441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:19.640402    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:41:19.640608 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:19.640456    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 10:41:20.640963 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:20.640901    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 10:41:20.641356 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:20.640984    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 10:41:21.645396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:21.645364    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:41:21.646719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:21.646684    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 10:41:24.796701 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.796664    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"]
Nov 03 10:41:24.800043 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.800021    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:24.808028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.808004    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"]
Nov 03 10:41:24.811220 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.810053    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"simulator-w2-e78c33-kserve-self-signed-certs\""
Nov 03 10:41:24.811384 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.810074    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"openshift-service-ca.crt\""
Nov 03 10:41:24.811384 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.810142    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"default-dockercfg-6dbqs\""
Nov 03 10:41:24.811519 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.810208    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"kube-root-ca.crt\""
Nov 03 10:41:24.932345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.932307    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-tmp-dir\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:24.932345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.932343    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64742526-7053-46b4-afd4-5043ebbda083-tls-certs\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:24.932555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.932371    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-home\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:24.932555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.932538    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-dshm\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:24.932619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.932569    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-model-cache\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:24.932619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:24.932599    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88w8\" (UniqueName: \"kubernetes.io/projected/64742526-7053-46b4-afd4-5043ebbda083-kube-api-access-k88w8\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.033844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.033810    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-tmp-dir\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.033844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.033848    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64742526-7053-46b4-afd4-5043ebbda083-tls-certs\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.034076 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.034021    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-home\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.034180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.034163    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-dshm\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.034226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.034200    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-model-cache\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.034271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.034236    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88w8\" (UniqueName: \"kubernetes.io/projected/64742526-7053-46b4-afd4-5043ebbda083-kube-api-access-k88w8\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.034271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.034263    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-tmp-dir\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.034347 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.034314    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-home\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.034455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.034435    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-model-cache\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.036612 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.036585    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-dshm\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.036779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.036761    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64742526-7053-46b4-afd4-5043ebbda083-tls-certs\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.042520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.042498    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88w8\" (UniqueName: \"kubernetes.io/projected/64742526-7053-46b4-afd4-5043ebbda083-kube-api-access-k88w8\") pod \"simulator-w2-e78c33-kserve-55679b5b88-ds4bb\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") " pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.113580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.113550    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:25.387903 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.387878    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"]
Nov 03 10:41:25.390715 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:41:25.390685    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64742526_7053_46b4_afd4_5043ebbda083.slice/crio-f75818b558e60ee9a2578f1c1c06c300038e74a0fbb11fe4c3a3d1680d5dbbae WatchSource:0}: Error finding container f75818b558e60ee9a2578f1c1c06c300038e74a0fbb11fe4c3a3d1680d5dbbae: Status 404 returned error can't find the container with id f75818b558e60ee9a2578f1c1c06c300038e74a0fbb11fe4c3a3d1680d5dbbae
Nov 03 10:41:25.546068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.546025    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" event={"ID":"64742526-7053-46b4-afd4-5043ebbda083","Type":"ContainerStarted","Data":"97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d"}
Nov 03 10:41:25.546227 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:25.546084    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" event={"ID":"64742526-7053-46b4-afd4-5043ebbda083","Type":"ContainerStarted","Data":"f75818b558e60ee9a2578f1c1c06c300038e74a0fbb11fe4c3a3d1680d5dbbae"}
Nov 03 10:41:26.114714 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:26.114681    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:32.744494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.744436    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podStartSLOduration=8.744421877 podStartE2EDuration="8.744421877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:25.562100347 +0000 UTC m=+729.466954789" watchObservedRunningTime="2026-11-03 10:41:32.744421877 +0000 UTC m=+736.649276317"
Nov 03 10:41:32.745525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.745506    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"]
Nov 03 10:41:32.749236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.749217    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:32.758411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.758161    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"default-dockercfg-lpsf5\""
Nov 03 10:41:32.758411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.758250    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"kube-root-ca.crt\""
Nov 03 10:41:32.758411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.758367    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"premium-w3-b1c6f6-kserve-self-signed-certs\""
Nov 03 10:41:32.758822 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.758796    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"]
Nov 03 10:41:32.759150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.759116    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"openshift-service-ca.crt\""
Nov 03 10:41:32.932538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.932491    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-tmp-dir\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:32.932839 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.932817    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-dshm\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:32.932986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.932883    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-model-cache\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:32.932986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.932957    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd7gx\" (UniqueName: \"kubernetes.io/projected/02beb4c4-a80f-493a-a421-ac714c3e256d-kube-api-access-qd7gx\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:32.933096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.933008    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-home\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:32.933096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:32.933086    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02beb4c4-a80f-493a-a421-ac714c3e256d-tls-certs\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034000    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-home\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034062    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02beb4c4-a80f-493a-a421-ac714c3e256d-tls-certs\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034107    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-tmp-dir\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034123    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-dshm\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034155    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-model-cache\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034186    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7gx\" (UniqueName: \"kubernetes.io/projected/02beb4c4-a80f-493a-a421-ac714c3e256d-kube-api-access-qd7gx\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034547 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034523    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-home\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034563    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-tmp-dir\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.034691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.034622    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-model-cache\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.037564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.037536    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-dshm\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.037796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.037770    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02beb4c4-a80f-493a-a421-ac714c3e256d-tls-certs\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.042627 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.042602    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7gx\" (UniqueName: \"kubernetes.io/projected/02beb4c4-a80f-493a-a421-ac714c3e256d-kube-api-access-qd7gx\") pod \"premium-w3-b1c6f6-kserve-b8fbcf987-4z44w\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.066513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.066490    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:33.342074 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.342052    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"]
Nov 03 10:41:33.344269 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:41:33.344238    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02beb4c4_a80f_493a_a421_ac714c3e256d.slice/crio-98eb41feb9d729cab1bf4b614407b9120ab95db07df8e74bf61bae82502cebed WatchSource:0}: Error finding container 98eb41feb9d729cab1bf4b614407b9120ab95db07df8e74bf61bae82502cebed: Status 404 returned error can't find the container with id 98eb41feb9d729cab1bf4b614407b9120ab95db07df8e74bf61bae82502cebed
Nov 03 10:41:33.595152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.595053    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" event={"ID":"02beb4c4-a80f-493a-a421-ac714c3e256d","Type":"ContainerStarted","Data":"68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5"}
Nov 03 10:41:33.595152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.595096    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" event={"ID":"02beb4c4-a80f-493a-a421-ac714c3e256d","Type":"ContainerStarted","Data":"98eb41feb9d729cab1bf4b614407b9120ab95db07df8e74bf61bae82502cebed"}
Nov 03 10:41:33.612135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:33.612076    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podStartSLOduration=1.6120613609999999 podStartE2EDuration="1.612061361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:33.610896839 +0000 UTC m=+737.515751281" watchObservedRunningTime="2026-11-03 10:41:33.612061361 +0000 UTC m=+737.516916016"
Nov 03 10:41:34.067224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:34.067190    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:35.114580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:35.114543    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:35.127528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:35.127493    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:35.616184 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:35.616153    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:41:40.739104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.739067    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"]
Nov 03 10:41:40.747223 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.747185    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.751470 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.751438    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"kube-root-ca.crt\""
Nov 03 10:41:40.752152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.752122    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"openshift-service-ca.crt\""
Nov 03 10:41:40.752769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.752741    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"]
Nov 03 10:41:40.753307 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.753282    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"default-dockercfg-rp5ss\""
Nov 03 10:41:40.754020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.753997    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f78621\"/\"premium-w0-f78621-kserve-self-signed-certs\""
Nov 03 10:41:40.820861 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.820819    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3b2439-ed34-4759-a416-95d44a62d63f-tls-certs\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.821087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.820954    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-model-cache\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.821087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.821003    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-dshm\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.821217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.821125    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-home\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.821217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.821153    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5xb\" (UniqueName: \"kubernetes.io/projected/9c3b2439-ed34-4759-a416-95d44a62d63f-kube-api-access-dt5xb\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.821383 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.821258    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-tmp-dir\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.921879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.921843    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3b2439-ed34-4759-a416-95d44a62d63f-tls-certs\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.921879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.921888    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-model-cache\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.922166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.921905    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-dshm\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.922166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.921980    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-home\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.922166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.922004    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5xb\" (UniqueName: \"kubernetes.io/projected/9c3b2439-ed34-4759-a416-95d44a62d63f-kube-api-access-dt5xb\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.922166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.922056    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-tmp-dir\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.922439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.922421    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-tmp-dir\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.922535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.922512    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-model-cache\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.922741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.922724    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-home\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.924901 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.924879    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-dshm\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.925226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.925205    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3b2439-ed34-4759-a416-95d44a62d63f-tls-certs\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:40.930317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:40.930289    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5xb\" (UniqueName: \"kubernetes.io/projected/9c3b2439-ed34-4759-a416-95d44a62d63f-kube-api-access-dt5xb\") pod \"premium-w0-f78621-kserve-7c9c554d9d-bm6h7\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") " pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:41.063842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:41.063757    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:41.337115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:41.337062    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"]
Nov 03 10:41:41.339988 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:41:41.339953    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c3b2439_ed34_4759_a416_95d44a62d63f.slice/crio-645c962aed55823bc829e74889c4867e9dbafd7b457876908a0e500fbd06a0c7 WatchSource:0}: Error finding container 645c962aed55823bc829e74889c4867e9dbafd7b457876908a0e500fbd06a0c7: Status 404 returned error can't find the container with id 645c962aed55823bc829e74889c4867e9dbafd7b457876908a0e500fbd06a0c7
Nov 03 10:41:41.636330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:41.636296    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" event={"ID":"9c3b2439-ed34-4759-a416-95d44a62d63f","Type":"ContainerStarted","Data":"cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4"}
Nov 03 10:41:41.636330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:41.636335    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" event={"ID":"9c3b2439-ed34-4759-a416-95d44a62d63f","Type":"ContainerStarted","Data":"645c962aed55823bc829e74889c4867e9dbafd7b457876908a0e500fbd06a0c7"}
Nov 03 10:41:41.655409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:41.655355    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podStartSLOduration=1.655339636 podStartE2EDuration="1.655339636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:41.654264446 +0000 UTC m=+745.559118890" watchObservedRunningTime="2026-11-03 10:41:41.655339636 +0000 UTC m=+745.560194101"
Nov 03 10:41:42.064355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:42.064102    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:43.066997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:43.066959    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:43.079868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:43.079842    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:43.188451 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:41:43.188415    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:41:43.659065 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:43.659037    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:41:44.852295 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.852261    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"]
Nov 03 10:41:44.855826 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.855806    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.865881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.865857    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e78c33-openshift-default-dockercfg-brqpb\""
Nov 03 10:41:44.867651 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.867630    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"]
Nov 03 10:41:44.984438 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984401    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.984642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984453    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.984642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984472    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.984642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984587    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.984642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984628    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.984834 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984650    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.984834 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984734    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.984834 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984764    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf54t\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-kube-api-access-bf54t\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:44.985016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:44.984869    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086177    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086228    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086347    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086384    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf54t\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-kube-api-access-bf54t\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086430    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086494    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086566    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086591    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086635    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.086757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.086672    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.087122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.087076    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.087221 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.087149    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.087285 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.087225    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.087654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.087626    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.089043 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.089008    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.089550 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.089534    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.095369 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.095346    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.098813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.096397    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf54t\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-kube-api-access-bf54t\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.170586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.170503    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:45.450296 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.450268    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"]
Nov 03 10:41:45.452754 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:41:45.452728    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663dc3ce_c558_4fc1_ab57_217b48ae1af8.slice/crio-6872ebc00f253772371c883cd69c68cdf25ad465fe61f83841102c3b7821c4c4 WatchSource:0}: Error finding container 6872ebc00f253772371c883cd69c68cdf25ad465fe61f83841102c3b7821c4c4: Status 404 returned error can't find the container with id 6872ebc00f253772371c883cd69c68cdf25ad465fe61f83841102c3b7821c4c4
Nov 03 10:41:45.455330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.455290    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:45.455419 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.455372    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:45.455476 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.455449    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:45.661337 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.661291    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" event={"ID":"663dc3ce-c558-4fc1-ab57-217b48ae1af8","Type":"ContainerStarted","Data":"2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13"}
Nov 03 10:41:45.661619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.661581    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" event={"ID":"663dc3ce-c558-4fc1-ab57-217b48ae1af8","Type":"ContainerStarted","Data":"6872ebc00f253772371c883cd69c68cdf25ad465fe61f83841102c3b7821c4c4"}
Nov 03 10:41:45.680763 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:45.680696    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podStartSLOduration=1.6806760330000001 podStartE2EDuration="1.680676033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:45.679810026 +0000 UTC m=+749.584664466" watchObservedRunningTime="2026-11-03 10:41:45.680676033 +0000 UTC m=+749.585530474"
Nov 03 10:41:46.171146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.171057    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:46.172850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.172822    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 03 10:41:46.172946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.172886    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 03 10:41:46.762370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.762332    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"]
Nov 03 10:41:46.778192 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.777320    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"]
Nov 03 10:41:46.778192 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.777433    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.783233 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.781382    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b1c6f6-openshift-default-dockercfg-h8q5b\""
Nov 03 10:41:46.921667 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.921630    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.921667 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.921669    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.921983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.921694    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.921983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.921843    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.921983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.921874    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.921983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.921892    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckcdq\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-kube-api-access-ckcdq\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.921983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.921945    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.922179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.922004    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:46.922179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:46.922055    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023515    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023598    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023616    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023635    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023666    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023688    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023708    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckcdq\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-kube-api-access-ckcdq\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023748    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.023850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.023776    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.024287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.024070    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.024287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.024157    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.024396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.024323    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.024396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.024351    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.024482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.024442    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.026616 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.026596    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.026745 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.026727    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.032276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.032255    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckcdq\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-kube-api-access-ckcdq\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.032276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.032263    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.116713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.116682    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:47.173026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.172749    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 03 10:41:47.173026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.172811    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 03 10:41:47.409575 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.409468    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"]
Nov 03 10:41:47.413954 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:41:47.413902    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9a3ed4_56b5_48cc_91ad_3bdc09cff226.slice/crio-152539bfbffab0ddd953a9023b51eea74c6304b9224bd326a82264cb3a44894e WatchSource:0}: Error finding container 152539bfbffab0ddd953a9023b51eea74c6304b9224bd326a82264cb3a44894e: Status 404 returned error can't find the container with id 152539bfbffab0ddd953a9023b51eea74c6304b9224bd326a82264cb3a44894e
Nov 03 10:41:47.416549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.416517    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:47.416650 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.416620    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:47.416710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.416664    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:47.673844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.673745    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" event={"ID":"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226","Type":"ContainerStarted","Data":"3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8"}
Nov 03 10:41:47.674017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.673845    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" event={"ID":"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226","Type":"ContainerStarted","Data":"152539bfbffab0ddd953a9023b51eea74c6304b9224bd326a82264cb3a44894e"}
Nov 03 10:41:47.695623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:47.693456    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podStartSLOduration=1.693437713 podStartE2EDuration="1.693437713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:47.692388989 +0000 UTC m=+751.597243433" watchObservedRunningTime="2026-11-03 10:41:47.693437713 +0000 UTC m=+751.598292160"
Nov 03 10:41:48.123735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:48.123700    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:48.131122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:48.131089    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 10:41:48.131368 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:48.131343    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 10:41:48.175482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:48.175448    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 03 10:41:48.176166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:48.176124    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 03 10:41:49.120471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:49.120434    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 10:41:49.120799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:49.120753    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 10:41:49.171907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:49.171863    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 03 10:41:49.172146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:49.171969    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 03 10:41:50.117947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:50.117450    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 10:41:50.117947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:50.117523    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 10:41:50.171375 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:50.171345    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 03 10:41:50.171595 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:50.171570    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 03 10:41:51.063938 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:51.063893    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:51.076467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:51.076441    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:51.133845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:51.133812    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 10:41:51.134396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:51.134355    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 10:41:51.172020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:51.171981    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 03 10:41:51.172231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:51.172038    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 03 10:41:51.729829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:51.729793    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:41:52.117110 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:52.117076    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 10:41:52.117287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:52.117134    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 10:41:52.175411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:52.175377    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:52.175886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:52.175835    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:52.176562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:52.176544    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 10:41:53.117591 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:53.117551    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 10:41:53.117749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:53.117607    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 10:41:54.327758 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:54.327730    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:54.328287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:54.328134    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:54.329030 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:54.329008    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 10:41:56.996743 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:56.996655    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"]
Nov 03 10:41:57.000642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.000619    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.005542 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.005520    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-010232-openshift-default-dockercfg-txh8q\""
Nov 03 10:41:57.013881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.013851    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"]
Nov 03 10:41:57.088059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088028    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c1b717-daba-4377-ad8d-5cc357d45428-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.088250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088066    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.088250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088193    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.088250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088228    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dngg\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-kube-api-access-5dngg\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.088397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088371    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.088455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088424    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.088502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088479    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.088536 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088499    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.088571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.088559    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c1b717-daba-4377-ad8d-5cc357d45428-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.189749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.189715    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c1b717-daba-4377-ad8d-5cc357d45428-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.189950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.189760    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c1b717-daba-4377-ad8d-5cc357d45428-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.189950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.189780    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.189950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.189859    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.189950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.189892    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dngg\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-kube-api-access-5dngg\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.189964    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.189990    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.190020    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.190043    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190356 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.190279    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.190512    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.190529    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.190606    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c1b717-daba-4377-ad8d-5cc357d45428-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.190703 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.190613    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.192374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.192354    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.192649 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.192630    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c1b717-daba-4377-ad8d-5cc357d45428-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.197867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.197847    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.197991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.197975    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dngg\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-kube-api-access-5dngg\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.314501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.314414    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:57.596342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.596287    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"]
Nov 03 10:41:57.598787 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:41:57.598733    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c1b717_daba_4377_ad8d_5cc357d45428.slice/crio-f715baac9e08a5a19e6a937bb58829ae731dbd2a46a28bed21eaf6e9effa8c96 WatchSource:0}: Error finding container f715baac9e08a5a19e6a937bb58829ae731dbd2a46a28bed21eaf6e9effa8c96: Status 404 returned error can't find the container with id f715baac9e08a5a19e6a937bb58829ae731dbd2a46a28bed21eaf6e9effa8c96
Nov 03 10:41:57.601342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.601300    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:57.601426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.601391    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:57.601426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.601437    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:41:57.763892 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.763841    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" event={"ID":"37c1b717-daba-4377-ad8d-5cc357d45428","Type":"ContainerStarted","Data":"1866da88411e647035ffa9008bfdfe35b0e12075836f1cf684dda8bb5a722868"}
Nov 03 10:41:57.764097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.763911    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" event={"ID":"37c1b717-daba-4377-ad8d-5cc357d45428","Type":"ContainerStarted","Data":"f715baac9e08a5a19e6a937bb58829ae731dbd2a46a28bed21eaf6e9effa8c96"}
Nov 03 10:41:57.787133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:57.787076    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" podStartSLOduration=1.787060759 podStartE2EDuration="1.787060759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:41:57.784660895 +0000 UTC m=+761.689515339" watchObservedRunningTime="2026-11-03 10:41:57.787060759 +0000 UTC m=+761.691915223"
Nov 03 10:41:58.315261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:58.315222    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:41:58.316885 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:58.316852    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:41:58.317071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:58.316929    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 10:41:59.315205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:59.315170    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:41:59.315386 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:41:59.315229    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 10:42:00.315582 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:00.315538    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:42:00.316076 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:00.315603    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 10:42:01.315691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:01.315652    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:42:01.316121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:01.315723    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 10:42:02.316355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:02.316317    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:42:02.317011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:02.316980    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 10:42:03.314844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:03.314808    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 10:42:03.315036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:03.314875    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 10:42:04.330538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:04.330508    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:42:04.331049 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:04.330950    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:42:04.331707 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:04.331688    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:42:13.211108 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:42:13.211070    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:42:28.697546 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.697503    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"]
Nov 03 10:42:28.702829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.702803    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.706580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.706559    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:42:28.706752 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.706677    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:42:28.708024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.708000    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"]
Nov 03 10:42:28.787457 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.787418    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9qc\" (UniqueName: \"kubernetes.io/projected/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-kube-api-access-dn9qc\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.787644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.787472    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.787644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.787620    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.888936 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.888882    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.889132 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.888956    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9qc\" (UniqueName: \"kubernetes.io/projected/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-kube-api-access-dn9qc\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.889132 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.888987    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.889686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.889662    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.891460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.891436    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:28.897470 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:28.897447    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9qc\" (UniqueName: \"kubernetes.io/projected/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-kube-api-access-dn9qc\") pod \"maas-api-revoke-sub-93d8feaf-npm9s\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:29.017567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:29.017494    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:29.313315 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:29.313288    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"]
Nov 03 10:42:29.315562 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:42:29.315531    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807c2e3e_ede1_4a38_a8f5_7c6e90eb8846.slice/crio-3ca9fe2d30aad53c3ee0940c0c01af1dec62739767de68946cff9df1f4b25d78 WatchSource:0}: Error finding container 3ca9fe2d30aad53c3ee0940c0c01af1dec62739767de68946cff9df1f4b25d78: Status 404 returned error can't find the container with id 3ca9fe2d30aad53c3ee0940c0c01af1dec62739767de68946cff9df1f4b25d78
Nov 03 10:42:30.033243 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:30.033207    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s" event={"ID":"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846","Type":"ContainerStarted","Data":"3dcbc3d54871b141678b87937b9dd0d83eb9de7f04c9fb9438a58a2ca8c6339d"}
Nov 03 10:42:30.033762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:30.033268    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s" event={"ID":"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846","Type":"ContainerStarted","Data":"3ca9fe2d30aad53c3ee0940c0c01af1dec62739767de68946cff9df1f4b25d78"}
Nov 03 10:42:30.053818 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:30.053757    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s" podStartSLOduration=2.053739373 podStartE2EDuration="2.053739373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:42:30.050462941 +0000 UTC m=+793.955317383" watchObservedRunningTime="2026-11-03 10:42:30.053739373 +0000 UTC m=+793.958593816"
Nov 03 10:42:31.040061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:31.040028    5393 generic.go:356] "Generic (PLEG): container finished" podID="807c2e3e-ede1-4a38-a8f5-7c6e90eb8846" containerID="3dcbc3d54871b141678b87937b9dd0d83eb9de7f04c9fb9438a58a2ca8c6339d" exitCode=0
Nov 03 10:42:31.040465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:31.040114    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s" event={"ID":"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846","Type":"ContainerDied","Data":"3dcbc3d54871b141678b87937b9dd0d83eb9de7f04c9fb9438a58a2ca8c6339d"}
Nov 03 10:42:32.174757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.174724    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:32.245465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.245433    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-openshift-ca\") pod \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") "
Nov 03 10:42:32.245644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.245566    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-xks-ca\") pod \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") "
Nov 03 10:42:32.245644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.245628    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-kube-api-access-dn9qc\" (UniqueName: \"kubernetes.io/projected/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-kube-api-access-dn9qc\") pod \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\" (UID: \"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846\") "
Nov 03 10:42:32.245781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.245756    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-openshift-ca" pod "807c2e3e-ede1-4a38-a8f5-7c6e90eb8846" (UID: "807c2e3e-ede1-4a38-a8f5-7c6e90eb8846"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:42:32.246021 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.245994    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:32.247824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.247794    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-xks-ca" pod "807c2e3e-ede1-4a38-a8f5-7c6e90eb8846" (UID: "807c2e3e-ede1-4a38-a8f5-7c6e90eb8846"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:42:32.247946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.247833    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-kube-api-access-dn9qc" pod "807c2e3e-ede1-4a38-a8f5-7c6e90eb8846" (UID: "807c2e3e-ede1-4a38-a8f5-7c6e90eb8846"). InnerVolumeSpecName "kube-api-access-dn9qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:42:32.346890 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.346861    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:32.346890 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:32.346891    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn9qc\" (UniqueName: \"kubernetes.io/projected/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846-kube-api-access-dn9qc\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:42:33.080212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:33.080175    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s" event={"ID":"807c2e3e-ede1-4a38-a8f5-7c6e90eb8846","Type":"ContainerDied","Data":"3ca9fe2d30aad53c3ee0940c0c01af1dec62739767de68946cff9df1f4b25d78"}
Nov 03 10:42:33.080411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:33.080233    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ca9fe2d30aad53c3ee0940c0c01af1dec62739767de68946cff9df1f4b25d78"
Nov 03 10:42:33.080411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:42:33.080296    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"
Nov 03 10:42:43.198337 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:42:43.198189    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:43:04.155822 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.155775    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"]
Nov 03 10:43:04.156572 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.156550    5393 cpu_manager.go:414] "containerMap: removing container" podUID="807c2e3e-ede1-4a38-a8f5-7c6e90eb8846" containerName="revoke-keys"
Nov 03 10:43:04.156695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.156575    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807c2e3e-ede1-4a38-a8f5-7c6e90eb8846" containerName="revoke-keys"
Nov 03 10:43:04.156757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.156711    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="807c2e3e-ede1-4a38-a8f5-7c6e90eb8846" containerName="revoke-keys"
Nov 03 10:43:04.160048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.160025    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.167135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.167099    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"e2e-conflict-a-3210672e-kserve-self-signed-certs\""
Nov 03 10:43:04.172257 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.171681    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"]
Nov 03 10:43:04.310586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.310538    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-model-cache\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.310878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.310860    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-dshm\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.311099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.311079    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-tmp-dir\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.311363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.311342    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-home\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.311489 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.311473    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97852767-b8b1-4a7b-83b1-51ef489d2151-tls-certs\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.311632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.311611    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ttx\" (UniqueName: \"kubernetes.io/projected/97852767-b8b1-4a7b-83b1-51ef489d2151-kube-api-access-x6ttx\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.412641 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.412555    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-model-cache\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.412641 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.412598    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-dshm\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.412866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.412657    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-tmp-dir\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.412866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.412764    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-home\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.412866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.412789    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97852767-b8b1-4a7b-83b1-51ef489d2151-tls-certs\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.413076 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.412956    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ttx\" (UniqueName: \"kubernetes.io/projected/97852767-b8b1-4a7b-83b1-51ef489d2151-kube-api-access-x6ttx\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.413129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.413080    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-model-cache\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.413200 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.413182    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-tmp-dir\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.413348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.413260    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-home\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.415314 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.415293    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-dshm\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.415648 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.415628    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97852767-b8b1-4a7b-83b1-51ef489d2151-tls-certs\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.424694 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.424665    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ttx\" (UniqueName: \"kubernetes.io/projected/97852767-b8b1-4a7b-83b1-51ef489d2151-kube-api-access-x6ttx\") pod \"e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.481029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.480992    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:04.819999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:04.818805    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"]
Nov 03 10:43:05.394045 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:05.394010    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn" event={"ID":"97852767-b8b1-4a7b-83b1-51ef489d2151","Type":"ContainerStarted","Data":"f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7"}
Nov 03 10:43:05.394467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:05.394054    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn" event={"ID":"97852767-b8b1-4a7b-83b1-51ef489d2151","Type":"ContainerStarted","Data":"2ecba48cac3f63fc2db8143ab3225a840ae85a7e56fefb9bcd97ed70ad6ec326"}
Nov 03 10:43:05.412362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:05.412309    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn" podStartSLOduration=1.412294007 podStartE2EDuration="1.412294007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:05.410168812 +0000 UTC m=+829.315023254" watchObservedRunningTime="2026-11-03 10:43:05.412294007 +0000 UTC m=+829.317148448"
Nov 03 10:43:05.481684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:05.481652    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:13.189332 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:43:13.189292    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:43:14.481353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:14.481318    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:14.495272 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:14.495242    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:14.511964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:14.511933    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:43:15.440137 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.438379    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"]
Nov 03 10:43:15.443352 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.443328    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.448192 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.448157    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e78c33\"/\"distinct-w2-e78c33-kserve-self-signed-certs\""
Nov 03 10:43:15.456863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.456838    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"]
Nov 03 10:43:15.501121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.501086    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-home\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.501605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.501168    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a119d8-1185-4b77-b7d4-9f445928fec8-tls-certs\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.501605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.501282    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-tmp-dir\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.501605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.501376    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7htzk\" (UniqueName: \"kubernetes.io/projected/86a119d8-1185-4b77-b7d4-9f445928fec8-kube-api-access-7htzk\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.501605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.501442    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-model-cache\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.501605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.501537    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-dshm\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.604588 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.604544    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-dshm\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.605779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.605754    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-home\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.606025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.606005    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a119d8-1185-4b77-b7d4-9f445928fec8-tls-certs\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.606166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.606152    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-tmp-dir\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.606353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.606336    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7htzk\" (UniqueName: \"kubernetes.io/projected/86a119d8-1185-4b77-b7d4-9f445928fec8-kube-api-access-7htzk\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.607937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.606499    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-model-cache\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.608246 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.606614    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-tmp-dir\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.624945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.606320    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-home\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.625191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.609483    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-model-cache\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.648948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.644281    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-dshm\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.648948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.645743    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a119d8-1185-4b77-b7d4-9f445928fec8-tls-certs\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.661708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.661632    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7htzk\" (UniqueName: \"kubernetes.io/projected/86a119d8-1185-4b77-b7d4-9f445928fec8-kube-api-access-7htzk\") pod \"distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") " pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:15.760772 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:15.760681    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:16.041095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.041056    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"]
Nov 03 10:43:16.043711 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:43:16.043667    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86a119d8_1185_4b77_b7d4_9f445928fec8.slice/crio-7be5c1d3f6f352ee0c74872d56bd00f1ecf47e7c2c21ac826260e42cd2f8b47c WatchSource:0}: Error finding container 7be5c1d3f6f352ee0c74872d56bd00f1ecf47e7c2c21ac826260e42cd2f8b47c: Status 404 returned error can't find the container with id 7be5c1d3f6f352ee0c74872d56bd00f1ecf47e7c2c21ac826260e42cd2f8b47c
Nov 03 10:43:16.510709 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.510666    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" event={"ID":"86a119d8-1185-4b77-b7d4-9f445928fec8","Type":"ContainerStarted","Data":"f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397"}
Nov 03 10:43:16.511174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.510721    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" event={"ID":"86a119d8-1185-4b77-b7d4-9f445928fec8","Type":"ContainerStarted","Data":"7be5c1d3f6f352ee0c74872d56bd00f1ecf47e7c2c21ac826260e42cd2f8b47c"}
Nov 03 10:43:16.528758 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.528688    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podStartSLOduration=1.528669999 podStartE2EDuration="1.528669999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:16.528310632 +0000 UTC m=+840.433165077" watchObservedRunningTime="2026-11-03 10:43:16.528669999 +0000 UTC m=+840.433524442"
Nov 03 10:43:16.761642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.761561    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:16.880937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.880885    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"]
Nov 03 10:43:16.885059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.885023    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.897573 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.897544    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"]
Nov 03 10:43:16.922425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.922392    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.922611 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.922438    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6b93a-430d-44ac-9d84-335e693275af-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.922678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.922610    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.922736 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.922665    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.922788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.922746    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6b93a-430d-44ac-9d84-335e693275af-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.922788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.922763    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtjd\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-kube-api-access-zjtjd\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.922890 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.922788    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.923007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.922985    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:16.923063 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:16.923040    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024519    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024565    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024601    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024630    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6b93a-430d-44ac-9d84-335e693275af-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024661    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024680    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024732    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6b93a-430d-44ac-9d84-335e693275af-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024759    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtjd\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-kube-api-access-zjtjd\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.024887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.024797    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.025193 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.025094    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.025243 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.025219    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.025243 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.025225    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.025341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.025256    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.025612 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.025591    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6b93a-430d-44ac-9d84-335e693275af-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.027201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.027180    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.027327 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.027307    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6b93a-430d-44ac-9d84-335e693275af-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.032626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.032600    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.032772 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.032666    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtjd\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-kube-api-access-zjtjd\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.201740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.201699    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:17.485494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.485465    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"]
Nov 03 10:43:17.487850 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:43:17.487813    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c6b93a_430d_44ac_9d84_335e693275af.slice/crio-48c891324b268ce6081b46d8ae42039c552dc777b60c1688b6e7adb03a0ade18 WatchSource:0}: Error finding container 48c891324b268ce6081b46d8ae42039c552dc777b60c1688b6e7adb03a0ade18: Status 404 returned error can't find the container with id 48c891324b268ce6081b46d8ae42039c552dc777b60c1688b6e7adb03a0ade18
Nov 03 10:43:17.490726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.490639    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:17.490838 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.490768    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:17.490838 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.490817    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:17.518582 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:17.518549    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" event={"ID":"03c6b93a-430d-44ac-9d84-335e693275af","Type":"ContainerStarted","Data":"48c891324b268ce6081b46d8ae42039c552dc777b60c1688b6e7adb03a0ade18"}
Nov 03 10:43:18.525576 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:18.525527    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" event={"ID":"03c6b93a-430d-44ac-9d84-335e693275af","Type":"ContainerStarted","Data":"b92e76d80c753dd695cbd9e7da2c6a69e7bd791fe0aef6a335d310674c7e1cbc"}
Nov 03 10:43:18.548127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:18.548084    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" podStartSLOduration=2.548071439 podStartE2EDuration="2.548071439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:18.546659377 +0000 UTC m=+842.451513808" watchObservedRunningTime="2026-11-03 10:43:18.548071439 +0000 UTC m=+842.452925879"
Nov 03 10:43:19.211946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:19.207283    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:19.212208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:19.212166    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 10:43:19.212295 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:19.212240    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 10:43:20.202944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:20.202355    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 10:43:20.202944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:20.202418    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 10:43:21.207615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:21.207549    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 10:43:21.208091 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:21.207606    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 10:43:22.205951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:22.205433    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 10:43:22.205951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:22.205498    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 10:43:23.205943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:23.202310    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 10:43:23.205943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:23.202378    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 10:43:24.202346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.202307    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 10:43:24.202565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.202358    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 10:43:24.580335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.579654    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"]
Nov 03 10:43:24.590169 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.590119    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.598431 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.598401    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"e2e-conflict-b-3210672e-kserve-self-signed-certs\""
Nov 03 10:43:24.617664 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.617635    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"]
Nov 03 10:43:24.665256 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.665221    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-tmp-dir\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.665460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.665270    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34abccb8-c76b-4354-ba9c-c1392fb6067b-tls-certs\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.665460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.665364    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-home\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.665460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.665415    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-model-cache\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.665637 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.665461    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprw9\" (UniqueName: \"kubernetes.io/projected/34abccb8-c76b-4354-ba9c-c1392fb6067b-kube-api-access-qprw9\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.665637 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.665518    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-dshm\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.767948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.767588    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-tmp-dir\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.767948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.767638    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34abccb8-c76b-4354-ba9c-c1392fb6067b-tls-certs\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.767948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.767683    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-home\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.767948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.767725    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-model-cache\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.767948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.767751    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprw9\" (UniqueName: \"kubernetes.io/projected/34abccb8-c76b-4354-ba9c-c1392fb6067b-kube-api-access-qprw9\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.767948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.767802    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-dshm\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.774940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.768657    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-home\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.774940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.768900    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-tmp-dir\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.774940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.769526    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-model-cache\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.810991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.800395    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34abccb8-c76b-4354-ba9c-c1392fb6067b-tls-certs\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.811418 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.811386    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-dshm\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.834992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.823774    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprw9\" (UniqueName: \"kubernetes.io/projected/34abccb8-c76b-4354-ba9c-c1392fb6067b-kube-api-access-qprw9\") pod \"e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") " pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:24.916425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:24.916380    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:25.205747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.205715    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"]
Nov 03 10:43:25.207374 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:43:25.207349    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34abccb8_c76b_4354_ba9c_c1392fb6067b.slice/crio-b94e9bc1168ee5b9efc9b567696d3f5664756d6f96e1c1917b2e2a932f5954bd WatchSource:0}: Error finding container b94e9bc1168ee5b9efc9b567696d3f5664756d6f96e1c1917b2e2a932f5954bd: Status 404 returned error can't find the container with id b94e9bc1168ee5b9efc9b567696d3f5664756d6f96e1c1917b2e2a932f5954bd
Nov 03 10:43:25.207721 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.207700    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:25.208119 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.208049    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:25.209102 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.209016    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:43:25.602549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.602251    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v" event={"ID":"34abccb8-c76b-4354-ba9c-c1392fb6067b","Type":"ContainerStarted","Data":"f9f5d6a5bf9906f8715e46ba6a83f2068d2b3bd1ac4ad421a68dd336942f1bc6"}
Nov 03 10:43:25.602549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.602299    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v" event={"ID":"34abccb8-c76b-4354-ba9c-c1392fb6067b","Type":"ContainerStarted","Data":"b94e9bc1168ee5b9efc9b567696d3f5664756d6f96e1c1917b2e2a932f5954bd"}
Nov 03 10:43:25.623762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.623704    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v" podStartSLOduration=1.623684349 podStartE2EDuration="1.623684349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:25.622402899 +0000 UTC m=+849.527257338" watchObservedRunningTime="2026-11-03 10:43:25.623684349 +0000 UTC m=+849.528538796"
Nov 03 10:43:25.763155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.762387    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:25.793002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.792977    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:25.916953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:25.916861    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:26.620234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:26.620149    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:43:31.904482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.904443    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"]
Nov 03 10:43:31.909105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.909080    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.918346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.918307    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"]
Nov 03 10:43:31.957244 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957197    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.957407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957249    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.957480 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957432    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.957523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957485    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.957651 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957634    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.957691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957670    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlbj\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-kube-api-access-wjlbj\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.957724 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957710    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.957779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957764    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:31.957837 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:31.957823    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.059624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.059581    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.060018 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.059990    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.060196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.060178    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlbj\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-kube-api-access-wjlbj\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.060326 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.060309    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.060432 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.060417    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.060550 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.060535    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.060711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.060695    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.060817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.060804    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.061086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.061066    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.074939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.062327    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.077787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.076014    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.078017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.076257    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.081937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.076935    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.082646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.077718    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.090962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.090888    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.115937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.096753    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlbj\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-kube-api-access-wjlbj\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.116190 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.100323    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.116305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.104500    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.234131 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.234048    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:32.676199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.667899    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"]
Nov 03 10:43:32.694078 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:43:32.694044    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c1f4ec_83c5_4332_a2c3_53360d2ab8e0.slice/crio-75038b2a584ab6d7500dfeea187cf2d410843987c2b7743b7192f22fb8eb24a1 WatchSource:0}: Error finding container 75038b2a584ab6d7500dfeea187cf2d410843987c2b7743b7192f22fb8eb24a1: Status 404 returned error can't find the container with id 75038b2a584ab6d7500dfeea187cf2d410843987c2b7743b7192f22fb8eb24a1
Nov 03 10:43:32.703095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.703056    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:32.703305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.703284    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:32.703461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:32.703443    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:33.709947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:33.706618    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" event={"ID":"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0","Type":"ContainerStarted","Data":"57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722"}
Nov 03 10:43:33.709947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:33.706710    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" event={"ID":"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0","Type":"ContainerStarted","Data":"75038b2a584ab6d7500dfeea187cf2d410843987c2b7743b7192f22fb8eb24a1"}
Nov 03 10:43:33.748946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:33.741404    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podStartSLOduration=2.741383933 podStartE2EDuration="2.741383933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:33.735393173 +0000 UTC m=+857.640247644" watchObservedRunningTime="2026-11-03 10:43:33.741383933 +0000 UTC m=+857.646238375"
Nov 03 10:43:34.236944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.236758    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 10:43:34.236944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.236821    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 10:43:34.258946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.254947    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:34.457944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.457024    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:34.511014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.508013    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn" podUID="97852767-b8b1-4a7b-83b1-51ef489d2151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 10:43:34.611047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.611001    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:34.659080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.659021    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:34.727040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.726997    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:43:34.922353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:34.922321    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:35.091974 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:35.089293    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:35.236442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:35.236345    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 10:43:35.236825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:35.236792    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 10:43:35.988277 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:35.988243    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:43:36.235080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:36.235046    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 10:43:36.235247 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:36.235108    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 10:43:37.234553 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:37.234515    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 10:43:37.235194 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:37.235159    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 10:43:38.234417 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:38.234383    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 10:43:38.234589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:38.234432    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 10:43:39.235105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:39.235069    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 10:43:39.235497 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:39.235127    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 10:43:40.235805 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:40.235759    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 03 10:43:40.236271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:40.235877    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 03 10:43:41.239183 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:41.239155    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:41.239580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:41.239438    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:41.240267 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:41.240251    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:43:43.218040 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:43:43.218008    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:43:45.068232 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.068184    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"]
Nov 03 10:43:45.072424 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.072398    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.089134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.083711    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"]
Nov 03 10:43:45.190175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190138    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.190385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190195    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.190385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190212    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.190385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190296    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.190385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190322    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf58613-07c6-4c00-98c2-697d8763ed79-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.190557 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190426    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.190557 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190473    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.190557 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190496    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.190695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.190626    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2b6\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-kube-api-access-9x2b6\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291469 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291428    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291645 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291476    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291645 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291539    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2b6\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-kube-api-access-9x2b6\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291652    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291690    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291713    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291742    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291768    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf58613-07c6-4c00-98c2-697d8763ed79-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291816    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.291994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.291947    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.292146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.292070    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.292146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.292100    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.292301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.292282    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.292662 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.292641    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf58613-07c6-4c00-98c2-697d8763ed79-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.294304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.294284    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.294429 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.294410    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.299373 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.299350    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.299743 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.299723    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2b6\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-kube-api-access-9x2b6\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.392989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.392911    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:45.699682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.698509    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"]
Nov 03 10:43:45.728970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.726778    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:45.728970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.726868    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:45.728970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.726911    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:43:45.913548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.913506    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" event={"ID":"cdf58613-07c6-4c00-98c2-697d8763ed79","Type":"ContainerStarted","Data":"4e9de5faae999a414abcbf52041f6a100abc45b72911376062ab338864c4b286"}
Nov 03 10:43:45.913728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.913560    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" event={"ID":"cdf58613-07c6-4c00-98c2-697d8763ed79","Type":"ContainerStarted","Data":"dec56472d66a8fa0f0bfe22d44fbebb41c30efe037fd19dba71f7d9e4dfd1837"}
Nov 03 10:43:45.935548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:45.935482    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podStartSLOduration=0.935462188 podStartE2EDuration="935.462188ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:45.933287283 +0000 UTC m=+869.838141723" watchObservedRunningTime="2026-11-03 10:43:45.935462188 +0000 UTC m=+869.840316630"
Nov 03 10:43:46.395985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:46.393273    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:46.400941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:46.399457    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 10:43:46.400941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:46.399521    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 10:43:47.393943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:47.393314    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 10:43:47.393943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:47.393374    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 10:43:48.401942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:48.394339    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 10:43:48.401942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:48.394404    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 10:43:49.351948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.350194    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"]
Nov 03 10:43:49.394946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.386198    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.411949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.411753    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"distinct-w1-010232-kserve-self-signed-certs\""
Nov 03 10:43:49.428946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.426417    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"]
Nov 03 10:43:49.436946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.435086    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 10:43:49.436946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.435166    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 10:43:49.514996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.512579    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-home\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.514996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.512724    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-tmp-dir\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.514996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.512781    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-dshm\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.514996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.512812    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-model-cache\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.514996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.512880    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtrrj\" (UniqueName: \"kubernetes.io/projected/0fe8b503-1baf-4082-b14c-6ee94460567e-kube-api-access-qtrrj\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.514996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.512983    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe8b503-1baf-4082-b14c-6ee94460567e-tls-certs\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.616904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.616800    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-tmp-dir\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.617678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.617656    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-dshm\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.618260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.618239    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-model-cache\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.618464 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.618444    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtrrj\" (UniqueName: \"kubernetes.io/projected/0fe8b503-1baf-4082-b14c-6ee94460567e-kube-api-access-qtrrj\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.619275 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.619255    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe8b503-1baf-4082-b14c-6ee94460567e-tls-certs\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.620039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.620020    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-home\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.620291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.617578    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-tmp-dir\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.620402 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.618832    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-model-cache\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.620735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.620718    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-home\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.657644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.657609    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtrrj\" (UniqueName: \"kubernetes.io/projected/0fe8b503-1baf-4082-b14c-6ee94460567e-kube-api-access-qtrrj\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.666509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.666477    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe8b503-1baf-4082-b14c-6ee94460567e-tls-certs\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.666727 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.666478    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-dshm\") pod \"distinct-w1-010232-kserve-6f46b7b64-hg7l8\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") " pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:49.755556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:49.755516    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:50.071527 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:50.071499    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"]
Nov 03 10:43:50.073621 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:43:50.073594    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fe8b503_1baf_4082_b14c_6ee94460567e.slice/crio-1ae9f9c7d270ad48048b8848a7e2a13ed1f699d69fc0f9d56bf0806eaeba824e WatchSource:0}: Error finding container 1ae9f9c7d270ad48048b8848a7e2a13ed1f699d69fc0f9d56bf0806eaeba824e: Status 404 returned error can't find the container with id 1ae9f9c7d270ad48048b8848a7e2a13ed1f699d69fc0f9d56bf0806eaeba824e
Nov 03 10:43:50.402945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:50.402359    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 10:43:50.402945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:50.402420    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 10:43:50.972851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:50.972812    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" event={"ID":"0fe8b503-1baf-4082-b14c-6ee94460567e","Type":"ContainerStarted","Data":"a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5"}
Nov 03 10:43:50.973316 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:50.972862    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" event={"ID":"0fe8b503-1baf-4082-b14c-6ee94460567e","Type":"ContainerStarted","Data":"1ae9f9c7d270ad48048b8848a7e2a13ed1f699d69fc0f9d56bf0806eaeba824e"}
Nov 03 10:43:51.003945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:50.997361    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podStartSLOduration=1.9973408780000002 podStartE2EDuration="1.997340878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:43:50.995421948 +0000 UTC m=+874.900276390" watchObservedRunningTime="2026-11-03 10:43:50.997340878 +0000 UTC m=+874.902195323"
Nov 03 10:43:51.395395 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:51.395333    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 10:43:51.395677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:51.395648    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 10:43:51.755991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:51.755893    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:52.393318 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:52.393284    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 10:43:52.393715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:52.393340    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 10:43:54.208649 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:54.208618    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:54.209251 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:54.209112    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:54.210389 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:54.210369    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 10:43:56.691008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:56.690954    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"]
Nov 03 10:43:59.756576 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:59.756540    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:43:59.775393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:43:59.775362    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:44:00.150794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:00.150766    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:44:01.349361 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:01.349300    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"]
Nov 03 10:44:12.045263 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.045177    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v" podUID="34abccb8-c76b-4354-ba9c-c1392fb6067b" containerName="main" containerID="cri-o://f9f5d6a5bf9906f8715e46ba6a83f2068d2b3bd1ac4ad421a68dd336942f1bc6" gracePeriod=45
Nov 03 10:44:12.268385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.268198    5393 generic.go:356] "Generic (PLEG): container finished" podID="34abccb8-c76b-4354-ba9c-c1392fb6067b" containerID="f9f5d6a5bf9906f8715e46ba6a83f2068d2b3bd1ac4ad421a68dd336942f1bc6" exitCode=0
Nov 03 10:44:12.268385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.268336    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v" event={"ID":"34abccb8-c76b-4354-ba9c-c1392fb6067b","Type":"ContainerDied","Data":"f9f5d6a5bf9906f8715e46ba6a83f2068d2b3bd1ac4ad421a68dd336942f1bc6"}
Nov 03 10:44:12.515382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.515356    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:44:12.679340 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679307    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34abccb8-c76b-4354-ba9c-c1392fb6067b-tls-certs\" (UniqueName: \"kubernetes.io/secret/34abccb8-c76b-4354-ba9c-c1392fb6067b-tls-certs\") pod \"34abccb8-c76b-4354-ba9c-c1392fb6067b\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") "
Nov 03 10:44:12.679531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679435    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-home\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-home\") pod \"34abccb8-c76b-4354-ba9c-c1392fb6067b\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") "
Nov 03 10:44:12.679531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679482    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-tmp-dir\") pod \"34abccb8-c76b-4354-ba9c-c1392fb6067b\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") "
Nov 03 10:44:12.679633 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679554    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-model-cache\") pod \"34abccb8-c76b-4354-ba9c-c1392fb6067b\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") "
Nov 03 10:44:12.679633 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679602    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-dshm\") pod \"34abccb8-c76b-4354-ba9c-c1392fb6067b\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") "
Nov 03 10:44:12.679733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679643    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34abccb8-c76b-4354-ba9c-c1392fb6067b-kube-api-access-qprw9\" (UniqueName: \"kubernetes.io/projected/34abccb8-c76b-4354-ba9c-c1392fb6067b-kube-api-access-qprw9\") pod \"34abccb8-c76b-4354-ba9c-c1392fb6067b\" (UID: \"34abccb8-c76b-4354-ba9c-c1392fb6067b\") "
Nov 03 10:44:12.679791 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679727    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-tmp-dir" pod "34abccb8-c76b-4354-ba9c-c1392fb6067b" (UID: "34abccb8-c76b-4354-ba9c-c1392fb6067b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:44:12.679791 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679702    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-home" pod "34abccb8-c76b-4354-ba9c-c1392fb6067b" (UID: "34abccb8-c76b-4354-ba9c-c1392fb6067b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:44:12.679791 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.679740    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-model-cache" pod "34abccb8-c76b-4354-ba9c-c1392fb6067b" (UID: "34abccb8-c76b-4354-ba9c-c1392fb6067b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:44:12.680067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.680048    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:12.680067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.680069    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:12.680264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.680085    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:12.682020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.681975    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34abccb8-c76b-4354-ba9c-c1392fb6067b-tls-certs" pod "34abccb8-c76b-4354-ba9c-c1392fb6067b" (UID: "34abccb8-c76b-4354-ba9c-c1392fb6067b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:12.682197 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.682173    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-dshm" pod "34abccb8-c76b-4354-ba9c-c1392fb6067b" (UID: "34abccb8-c76b-4354-ba9c-c1392fb6067b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:44:12.682367 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.682342    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34abccb8-c76b-4354-ba9c-c1392fb6067b-kube-api-access-qprw9" pod "34abccb8-c76b-4354-ba9c-c1392fb6067b" (UID: "34abccb8-c76b-4354-ba9c-c1392fb6067b"). InnerVolumeSpecName "kube-api-access-qprw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:12.781341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.781293    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34abccb8-c76b-4354-ba9c-c1392fb6067b-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:12.781341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.781340    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qprw9\" (UniqueName: \"kubernetes.io/projected/34abccb8-c76b-4354-ba9c-c1392fb6067b-kube-api-access-qprw9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:12.781558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:12.781356    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34abccb8-c76b-4354-ba9c-c1392fb6067b-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:13.189171 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:44:13.189131    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:44:13.275399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:13.275368    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"
Nov 03 10:44:13.275399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:13.275373    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v" event={"ID":"34abccb8-c76b-4354-ba9c-c1392fb6067b","Type":"ContainerDied","Data":"b94e9bc1168ee5b9efc9b567696d3f5664756d6f96e1c1917b2e2a932f5954bd"}
Nov 03 10:44:13.275648 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:13.275431    5393 scope.go:122] "RemoveContainer" containerID="f9f5d6a5bf9906f8715e46ba6a83f2068d2b3bd1ac4ad421a68dd336942f1bc6"
Nov 03 10:44:13.300612 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:13.300581    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"]
Nov 03 10:44:13.303792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:13.303766    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-b-3210672e-kserve-8bf84f7d7-x7n2v"]
Nov 03 10:44:14.075617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.075583    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"]
Nov 03 10:44:14.076365 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.076329    5393 cpu_manager.go:414] "containerMap: removing container" podUID="34abccb8-c76b-4354-ba9c-c1392fb6067b" containerName="main"
Nov 03 10:44:14.076537 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.076521    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34abccb8-c76b-4354-ba9c-c1392fb6067b" containerName="main"
Nov 03 10:44:14.076795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.076780    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="34abccb8-c76b-4354-ba9c-c1392fb6067b" containerName="main"
Nov 03 10:44:14.081957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.081934    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.088742 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.088711    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"]
Nov 03 10:44:14.091752 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.091730    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010232\"/\"distinct-2-w1-010232-kserve-self-signed-certs\""
Nov 03 10:44:14.197943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.197524    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21f137d-3ce4-49d8-9175-63b51e04f80c-tls-certs\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.197943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.197587    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9fzw\" (UniqueName: \"kubernetes.io/projected/d21f137d-3ce4-49d8-9175-63b51e04f80c-kube-api-access-z9fzw\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.197943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.197635    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-home\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.197943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.197700    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-model-cache\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.197943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.197736    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-dshm\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.197943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.197829    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-tmp-dir\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.247965 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.244813    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34abccb8-c76b-4354-ba9c-c1392fb6067b" path="/var/lib/kubelet/pods/34abccb8-c76b-4354-ba9c-c1392fb6067b/volumes"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.301621    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-tmp-dir\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.301766    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21f137d-3ce4-49d8-9175-63b51e04f80c-tls-certs\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.301806    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9fzw\" (UniqueName: \"kubernetes.io/projected/d21f137d-3ce4-49d8-9175-63b51e04f80c-kube-api-access-z9fzw\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.301842    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-home\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.301932    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-model-cache\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.301965    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-dshm\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.303046    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-tmp-dir\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.303615    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-home\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.303942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.303836    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-model-cache\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.309941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.306191    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-dshm\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.325985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.315191    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21f137d-3ce4-49d8-9175-63b51e04f80c-tls-certs\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.328961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.327358    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9fzw\" (UniqueName: \"kubernetes.io/projected/d21f137d-3ce4-49d8-9175-63b51e04f80c-kube-api-access-z9fzw\") pod \"distinct-2-w1-010232-kserve-8668d84b59-r62q5\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") " pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.400823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.400771    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:14.708653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:14.708627    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"]
Nov 03 10:44:14.711357 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:44:14.711328    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd21f137d_3ce4_49d8_9175_63b51e04f80c.slice/crio-8bd78084d21388e5bb1728687ea46618e8ccfd5368cccc6006a92785498424ec WatchSource:0}: Error finding container 8bd78084d21388e5bb1728687ea46618e8ccfd5368cccc6006a92785498424ec: Status 404 returned error can't find the container with id 8bd78084d21388e5bb1728687ea46618e8ccfd5368cccc6006a92785498424ec
Nov 03 10:44:15.298115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:15.298082    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" event={"ID":"d21f137d-3ce4-49d8-9175-63b51e04f80c","Type":"ContainerStarted","Data":"60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4"}
Nov 03 10:44:15.298553 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:15.298124    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" event={"ID":"d21f137d-3ce4-49d8-9175-63b51e04f80c","Type":"ContainerStarted","Data":"8bd78084d21388e5bb1728687ea46618e8ccfd5368cccc6006a92785498424ec"}
Nov 03 10:44:15.317323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:15.317267    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podStartSLOduration=1.317248628 podStartE2EDuration="1.317248628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:15.315698581 +0000 UTC m=+899.220553023" watchObservedRunningTime="2026-11-03 10:44:15.317248628 +0000 UTC m=+899.222103070"
Nov 03 10:44:15.401369 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:15.401331    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:16.345691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.345657    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:44:16.348798 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.348782    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:44:16.376746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.376704    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn" podUID="97852767-b8b1-4a7b-83b1-51ef489d2151" containerName="main" containerID="cri-o://f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7" gracePeriod=45
Nov 03 10:44:16.499321 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.499278    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn" podUID="97852767-b8b1-4a7b-83b1-51ef489d2151" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused"
Nov 03 10:44:16.695022 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.694994    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:44:16.766321 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766267    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-tmp-dir\") pod \"97852767-b8b1-4a7b-83b1-51ef489d2151\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") "
Nov 03 10:44:16.766522 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766335    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-model-cache\") pod \"97852767-b8b1-4a7b-83b1-51ef489d2151\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") "
Nov 03 10:44:16.766522 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766362    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-dshm\") pod \"97852767-b8b1-4a7b-83b1-51ef489d2151\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") "
Nov 03 10:44:16.766522 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766442    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-home\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-home\") pod \"97852767-b8b1-4a7b-83b1-51ef489d2151\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") "
Nov 03 10:44:16.766522 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766496    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97852767-b8b1-4a7b-83b1-51ef489d2151-kube-api-access-x6ttx\" (UniqueName: \"kubernetes.io/projected/97852767-b8b1-4a7b-83b1-51ef489d2151-kube-api-access-x6ttx\") pod \"97852767-b8b1-4a7b-83b1-51ef489d2151\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") "
Nov 03 10:44:16.766754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766540    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97852767-b8b1-4a7b-83b1-51ef489d2151-tls-certs\" (UniqueName: \"kubernetes.io/secret/97852767-b8b1-4a7b-83b1-51ef489d2151-tls-certs\") pod \"97852767-b8b1-4a7b-83b1-51ef489d2151\" (UID: \"97852767-b8b1-4a7b-83b1-51ef489d2151\") "
Nov 03 10:44:16.766754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766546    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-tmp-dir" pod "97852767-b8b1-4a7b-83b1-51ef489d2151" (UID: "97852767-b8b1-4a7b-83b1-51ef489d2151"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:44:16.766754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766649    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-model-cache" pod "97852767-b8b1-4a7b-83b1-51ef489d2151" (UID: "97852767-b8b1-4a7b-83b1-51ef489d2151"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:44:16.766754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766703    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-home" pod "97852767-b8b1-4a7b-83b1-51ef489d2151" (UID: "97852767-b8b1-4a7b-83b1-51ef489d2151"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:44:16.766986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766943    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:16.766986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766967    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:16.766986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.766981    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:16.768757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.768730    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-dshm" pod "97852767-b8b1-4a7b-83b1-51ef489d2151" (UID: "97852767-b8b1-4a7b-83b1-51ef489d2151"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:44:16.768865 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.768756    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97852767-b8b1-4a7b-83b1-51ef489d2151-kube-api-access-x6ttx" pod "97852767-b8b1-4a7b-83b1-51ef489d2151" (UID: "97852767-b8b1-4a7b-83b1-51ef489d2151"). InnerVolumeSpecName "kube-api-access-x6ttx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:16.769066 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.769051    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97852767-b8b1-4a7b-83b1-51ef489d2151-tls-certs" pod "97852767-b8b1-4a7b-83b1-51ef489d2151" (UID: "97852767-b8b1-4a7b-83b1-51ef489d2151"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:16.877975 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.875550    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97852767-b8b1-4a7b-83b1-51ef489d2151-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:16.877975 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.875588    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97852767-b8b1-4a7b-83b1-51ef489d2151-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:16.877975 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:16.875605    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ttx\" (UniqueName: \"kubernetes.io/projected/97852767-b8b1-4a7b-83b1-51ef489d2151-kube-api-access-x6ttx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:17.309371 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.309344    5393 generic.go:356] "Generic (PLEG): container finished" podID="97852767-b8b1-4a7b-83b1-51ef489d2151" containerID="f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7" exitCode=0
Nov 03 10:44:17.309551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.309382    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"
Nov 03 10:44:17.309551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.309433    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn" event={"ID":"97852767-b8b1-4a7b-83b1-51ef489d2151","Type":"ContainerDied","Data":"f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7"}
Nov 03 10:44:17.309551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.309480    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn" event={"ID":"97852767-b8b1-4a7b-83b1-51ef489d2151","Type":"ContainerDied","Data":"2ecba48cac3f63fc2db8143ab3225a840ae85a7e56fefb9bcd97ed70ad6ec326"}
Nov 03 10:44:17.309551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.309500    5393 scope.go:122] "RemoveContainer" containerID="f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7"
Nov 03 10:44:17.319994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.319974    5393 scope.go:122] "RemoveContainer" containerID="f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7"
Nov 03 10:44:17.320273 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:44:17.320254    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7\": container with ID starting with f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7 not found: ID does not exist" containerID="f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7"
Nov 03 10:44:17.320353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.320282    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7"} err="failed to get container status \"f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7\": rpc error: code = NotFound desc = could not find container \"f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7\": container with ID starting with f8f530bee112fb70dc6821c2868aaa843e2a3f211d2a061b00a4b9245862cfa7 not found: ID does not exist"
Nov 03 10:44:17.333313 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.333218    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"]
Nov 03 10:44:17.339153 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.339127    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/e2e-conflict-a-3210672e-kserve-84b4c49ff7-x4xcn"]
Nov 03 10:44:17.711621 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.711585    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"]
Nov 03 10:44:17.712152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.712125    5393 cpu_manager.go:414] "containerMap: removing container" podUID="97852767-b8b1-4a7b-83b1-51ef489d2151" containerName="main"
Nov 03 10:44:17.712152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.712142    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97852767-b8b1-4a7b-83b1-51ef489d2151" containerName="main"
Nov 03 10:44:17.712292 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.712257    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="97852767-b8b1-4a7b-83b1-51ef489d2151" containerName="main"
Nov 03 10:44:17.716816 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.716794    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.723565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.722548    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:44:17.723565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.722691    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:44:17.723565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.723268    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"]
Nov 03 10:44:17.793212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.793173    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.793413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.793244    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scn6\" (UniqueName: \"kubernetes.io/projected/1becc992-f950-4c6c-b71f-7f06e513f067-kube-api-access-2scn6\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.793485 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.793407    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.895156 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.895121    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.895370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.895270    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.895370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.895348    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scn6\" (UniqueName: \"kubernetes.io/projected/1becc992-f950-4c6c-b71f-7f06e513f067-kube-api-access-2scn6\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.896009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.895979    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.897730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.897710    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:17.903029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:17.903010    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scn6\" (UniqueName: \"kubernetes.io/projected/1becc992-f950-4c6c-b71f-7f06e513f067-kube-api-access-2scn6\") pod \"maas-api-revoke-sub-35a91942-lgsl2\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:18.035009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:18.034901    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:18.229596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:18.229561    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97852767-b8b1-4a7b-83b1-51ef489d2151" path="/var/lib/kubelet/pods/97852767-b8b1-4a7b-83b1-51ef489d2151/volumes"
Nov 03 10:44:18.312200 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:18.312173    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"]
Nov 03 10:44:18.313829 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:44:18.313798    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1becc992_f950_4c6c_b71f_7f06e513f067.slice/crio-6280447dda9df24a610af521d0fa03d71a244707ed58a1675b6f593c9625c05e WatchSource:0}: Error finding container 6280447dda9df24a610af521d0fa03d71a244707ed58a1675b6f593c9625c05e: Status 404 returned error can't find the container with id 6280447dda9df24a610af521d0fa03d71a244707ed58a1675b6f593c9625c05e
Nov 03 10:44:19.324438 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:19.324405    5393 generic.go:356] "Generic (PLEG): container finished" podID="1becc992-f950-4c6c-b71f-7f06e513f067" containerID="0028f67ce55d2eccb242f16761ce1d51119bc30fce4a0e1a4dda30bb3bff315a" exitCode=0
Nov 03 10:44:19.324825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:19.324487    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2" event={"ID":"1becc992-f950-4c6c-b71f-7f06e513f067","Type":"ContainerDied","Data":"0028f67ce55d2eccb242f16761ce1d51119bc30fce4a0e1a4dda30bb3bff315a"}
Nov 03 10:44:19.324825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:19.324536    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2" event={"ID":"1becc992-f950-4c6c-b71f-7f06e513f067","Type":"ContainerStarted","Data":"6280447dda9df24a610af521d0fa03d71a244707ed58a1675b6f593c9625c05e"}
Nov 03 10:44:20.484564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.484538    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:20.625197 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.625165    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-xks-ca\") pod \"1becc992-f950-4c6c-b71f-7f06e513f067\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") "
Nov 03 10:44:20.625351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.625238    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-openshift-ca\") pod \"1becc992-f950-4c6c-b71f-7f06e513f067\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") "
Nov 03 10:44:20.625427 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.625346    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1becc992-f950-4c6c-b71f-7f06e513f067-kube-api-access-2scn6\" (UniqueName: \"kubernetes.io/projected/1becc992-f950-4c6c-b71f-7f06e513f067-kube-api-access-2scn6\") pod \"1becc992-f950-4c6c-b71f-7f06e513f067\" (UID: \"1becc992-f950-4c6c-b71f-7f06e513f067\") "
Nov 03 10:44:20.625593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.625573    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-openshift-ca" pod "1becc992-f950-4c6c-b71f-7f06e513f067" (UID: "1becc992-f950-4c6c-b71f-7f06e513f067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:20.625912 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.625878    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:20.627545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.627520    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-xks-ca" pod "1becc992-f950-4c6c-b71f-7f06e513f067" (UID: "1becc992-f950-4c6c-b71f-7f06e513f067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:20.627660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.627644    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1becc992-f950-4c6c-b71f-7f06e513f067-kube-api-access-2scn6" pod "1becc992-f950-4c6c-b71f-7f06e513f067" (UID: "1becc992-f950-4c6c-b71f-7f06e513f067"). InnerVolumeSpecName "kube-api-access-2scn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:20.727526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.727488    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1becc992-f950-4c6c-b71f-7f06e513f067-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:20.727526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:20.727523    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2scn6\" (UniqueName: \"kubernetes.io/projected/1becc992-f950-4c6c-b71f-7f06e513f067-kube-api-access-2scn6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:21.337281 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:21.337246    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2" event={"ID":"1becc992-f950-4c6c-b71f-7f06e513f067","Type":"ContainerDied","Data":"6280447dda9df24a610af521d0fa03d71a244707ed58a1675b6f593c9625c05e"}
Nov 03 10:44:21.337465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:21.337298    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6280447dda9df24a610af521d0fa03d71a244707ed58a1675b6f593c9625c05e"
Nov 03 10:44:21.337465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:21.337252    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"
Nov 03 10:44:24.403871 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:24.403836    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:24.447768 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:24.447733    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:25.262086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:25.262046    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:44:25.262410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:25.262362    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:25.382051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:25.375289    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:44:25.382051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:25.375350    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:25.513570 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:25.513483    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:26.296279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.296246    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"]
Nov 03 10:44:26.297018 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.296997    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1becc992-f950-4c6c-b71f-7f06e513f067" containerName="revoke-keys"
Nov 03 10:44:26.297107 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.297021    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1becc992-f950-4c6c-b71f-7f06e513f067" containerName="revoke-keys"
Nov 03 10:44:26.297179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.297166    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1becc992-f950-4c6c-b71f-7f06e513f067" containerName="revoke-keys"
Nov 03 10:44:26.300636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.300617    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.305468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.305440    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"]
Nov 03 10:44:26.308698 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.308677    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b1c6f6\"/\"distinct-2-w3-b1c6f6-kserve-self-signed-certs\""
Nov 03 10:44:26.399715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.399684    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-home\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.399937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.399730    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-dshm\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.400012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.399934    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpln9\" (UniqueName: \"kubernetes.io/projected/ea895738-0cc7-4285-997a-35097600664d-kube-api-access-xpln9\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.400012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.399972    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-tmp-dir\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.400012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.399998    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-model-cache\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.400175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.400028    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea895738-0cc7-4285-997a-35097600664d-tls-certs\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501105    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-dshm\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501210    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpln9\" (UniqueName: \"kubernetes.io/projected/ea895738-0cc7-4285-997a-35097600664d-kube-api-access-xpln9\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501244    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-tmp-dir\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501271    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-model-cache\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501317    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea895738-0cc7-4285-997a-35097600664d-tls-certs\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501454    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-home\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501707 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501671    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-tmp-dir\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501804 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501781    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-home\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.501872 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.501832    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-model-cache\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.503728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.503703    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-dshm\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.504475 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.504453    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea895738-0cc7-4285-997a-35097600664d-tls-certs\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.510183 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.510158    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpln9\" (UniqueName: \"kubernetes.io/projected/ea895738-0cc7-4285-997a-35097600664d-kube-api-access-xpln9\") pod \"distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") " pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.623528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.623150    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:26.966516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:26.966487    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"]
Nov 03 10:44:26.969115 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:44:26.969081    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea895738_0cc7_4285_997a_35097600664d.slice/crio-d215a36c9f102418ea01adf12d20e74c527f1754a14699732c32e51120720cdd WatchSource:0}: Error finding container d215a36c9f102418ea01adf12d20e74c527f1754a14699732c32e51120720cdd: Status 404 returned error can't find the container with id d215a36c9f102418ea01adf12d20e74c527f1754a14699732c32e51120720cdd
Nov 03 10:44:27.236865 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.236780    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"]
Nov 03 10:44:27.241583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.241555    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.253237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.253204    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"]
Nov 03 10:44:27.312559 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.312514    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.312750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.312561    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.312750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.312593    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.312939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.312760    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.312939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.312888    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-kube-api-access-vv7jl\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.313047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.313009    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.313047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.313040    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.313134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.313084    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.313134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.313114    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.414637    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.414699    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.414750    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.414783    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.414846    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.414892    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.414940    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.415013    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.418061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.415120    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-kube-api-access-vv7jl\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.435614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.434467    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.435614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.434701    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.435614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.434744    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.435614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.434978    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.436614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.436590    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.458199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.444296    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.458199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.446656    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.458199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.448459    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-kube-api-access-vv7jl\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.458199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.448836    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.458199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.456094    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" event={"ID":"ea895738-0cc7-4285-997a-35097600664d","Type":"ContainerStarted","Data":"6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb"}
Nov 03 10:44:27.458199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.456138    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" event={"ID":"ea895738-0cc7-4285-997a-35097600664d","Type":"ContainerStarted","Data":"d215a36c9f102418ea01adf12d20e74c527f1754a14699732c32e51120720cdd"}
Nov 03 10:44:27.478023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.477970    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podStartSLOduration=1.4779542430000001 podStartE2EDuration="1.477954243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:27.476522487 +0000 UTC m=+911.381376964" watchObservedRunningTime="2026-11-03 10:44:27.477954243 +0000 UTC m=+911.382808686"
Nov 03 10:44:27.560574 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.560490    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:27.645431 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.645394    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:27.920963 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.920912    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"]
Nov 03 10:44:27.924504 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:44:27.924475    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee3e75b_fb64_4bf0_86cc_df7dfc4a2690.slice/crio-dc881e4a757b969308405023cebb51e8e2df99145373737c210eebf23ef95628 WatchSource:0}: Error finding container dc881e4a757b969308405023cebb51e8e2df99145373737c210eebf23ef95628: Status 404 returned error can't find the container with id dc881e4a757b969308405023cebb51e8e2df99145373737c210eebf23ef95628
Nov 03 10:44:27.927172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.927136    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:27.929175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.927216    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:27.929175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:27.927615    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:28.201306 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.201220    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"]
Nov 03 10:44:28.205289 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.205264    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.209442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.209390    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:44:28.209442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.209404    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:44:28.212805 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.212779    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"]
Nov 03 10:44:28.332741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.332702    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.333085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.333062    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2vg\" (UniqueName: \"kubernetes.io/projected/3102e53e-32ab-4b1b-9341-5f6faf806cc3-kube-api-access-fb2vg\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.333390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.333371    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.438608 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.437635    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.438608 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.437742    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2vg\" (UniqueName: \"kubernetes.io/projected/3102e53e-32ab-4b1b-9341-5f6faf806cc3-kube-api-access-fb2vg\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.438608 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.437901    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.439714 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.439679    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.443949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.442996    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.447723 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.447701    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2vg\" (UniqueName: \"kubernetes.io/projected/3102e53e-32ab-4b1b-9341-5f6faf806cc3-kube-api-access-fb2vg\") pod \"maas-api-revoke-sub-077cee3e-zj595\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.464017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.463938    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" event={"ID":"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690","Type":"ContainerStarted","Data":"39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492"}
Nov 03 10:44:28.464017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.463994    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" event={"ID":"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690","Type":"ContainerStarted","Data":"dc881e4a757b969308405023cebb51e8e2df99145373737c210eebf23ef95628"}
Nov 03 10:44:28.508072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.507391    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podStartSLOduration=1.5073691930000002 podStartE2EDuration="1.507369193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:28.504470062 +0000 UTC m=+912.409324504" watchObservedRunningTime="2026-11-03 10:44:28.507369193 +0000 UTC m=+912.412223613"
Nov 03 10:44:28.523609 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.523581    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:28.561447 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.561416    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:28.562903 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.562879    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 10:44:28.563045 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.562949    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 10:44:28.822994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:28.822104    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"]
Nov 03 10:44:29.471508 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:29.471420    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595" event={"ID":"3102e53e-32ab-4b1b-9341-5f6faf806cc3","Type":"ContainerStarted","Data":"85bc231166085ac91993076a26a895cfa69e6ffd1ad8b53a6e8738e649b5cf62"}
Nov 03 10:44:29.471686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:29.471516    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595" event={"ID":"3102e53e-32ab-4b1b-9341-5f6faf806cc3","Type":"ContainerStarted","Data":"dbde45d906f16dd42c1ffa6b3f2b135f05c9355f2d47af7a9c5e8f1eaea4d115"}
Nov 03 10:44:29.507942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:29.507497    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595" podStartSLOduration=1.507477056 podStartE2EDuration="1.507477056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:29.505165072 +0000 UTC m=+913.410019514" watchObservedRunningTime="2026-11-03 10:44:29.507477056 +0000 UTC m=+913.412331499"
Nov 03 10:44:29.561460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:29.561426    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 10:44:29.561621 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:29.561481    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 10:44:30.120906 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.119789    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"]
Nov 03 10:44:30.136205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.136126    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"]
Nov 03 10:44:30.136343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.136274    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174161    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174288    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174357    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174446    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174486    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174577 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174563    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdw47\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-kube-api-access-sdw47\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174634 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174605    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174653    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.174678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.174676    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.275837 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.275798    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.275865    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.275956    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.275986    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdw47\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-kube-api-access-sdw47\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276018    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276043    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276067    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276136    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276231    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276350    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-credential-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276370    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-socket\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276560    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-data\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.276711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276598    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-certs\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.277012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.276991    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istiod-ca-cert\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.278492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.278474    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-envoy\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.278763 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.278742    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-podinfo\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.316459 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.316427    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdw47\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-kube-api-access-sdw47\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.317300 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.317278    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-token\") pod \"e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") " pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.455400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.455314    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:30.478852 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.478822    5393 generic.go:356] "Generic (PLEG): container finished" podID="3102e53e-32ab-4b1b-9341-5f6faf806cc3" containerID="85bc231166085ac91993076a26a895cfa69e6ffd1ad8b53a6e8738e649b5cf62" exitCode=0
Nov 03 10:44:30.478987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.478899    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595" event={"ID":"3102e53e-32ab-4b1b-9341-5f6faf806cc3","Type":"ContainerDied","Data":"85bc231166085ac91993076a26a895cfa69e6ffd1ad8b53a6e8738e649b5cf62"}
Nov 03 10:44:30.561575 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.561540    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 10:44:30.561720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.561601    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 10:44:30.767302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.767276    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"]
Nov 03 10:44:30.769256 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:44:30.769225    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae62e87_8603_4fc9_8c0a_0a751df1ce7a.slice/crio-d405b1dbcdf25b191659c1c3acdc6245761fbbe3041a76084f73acd36fd25788 WatchSource:0}: Error finding container d405b1dbcdf25b191659c1c3acdc6245761fbbe3041a76084f73acd36fd25788: Status 404 returned error can't find the container with id d405b1dbcdf25b191659c1c3acdc6245761fbbe3041a76084f73acd36fd25788
Nov 03 10:44:30.771749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.771720    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:30.771814 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.771787    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:30.771851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:30.771816    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:31.502998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.502967    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" event={"ID":"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a","Type":"ContainerStarted","Data":"6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc"}
Nov 03 10:44:31.503646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.503604    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" event={"ID":"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a","Type":"ContainerStarted","Data":"d405b1dbcdf25b191659c1c3acdc6245761fbbe3041a76084f73acd36fd25788"}
Nov 03 10:44:31.539955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.538121    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" podStartSLOduration=1.53810291 podStartE2EDuration="1.53810291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:31.532761605 +0000 UTC m=+915.437616044" watchObservedRunningTime="2026-11-03 10:44:31.53810291 +0000 UTC m=+915.442957353"
Nov 03 10:44:31.561444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.561405    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 10:44:31.561604 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.561472    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 10:44:31.788962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.786194    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:31.851203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.850745    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-xks-ca\") pod \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") "
Nov 03 10:44:31.851203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.850793    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-openshift-ca\") pod \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") "
Nov 03 10:44:31.851203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.850837    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3102e53e-32ab-4b1b-9341-5f6faf806cc3-kube-api-access-fb2vg\" (UniqueName: \"kubernetes.io/projected/3102e53e-32ab-4b1b-9341-5f6faf806cc3-kube-api-access-fb2vg\") pod \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\" (UID: \"3102e53e-32ab-4b1b-9341-5f6faf806cc3\") "
Nov 03 10:44:31.851497 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.851406    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-openshift-ca" pod "3102e53e-32ab-4b1b-9341-5f6faf806cc3" (UID: "3102e53e-32ab-4b1b-9341-5f6faf806cc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:31.853578 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.853543    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3102e53e-32ab-4b1b-9341-5f6faf806cc3-kube-api-access-fb2vg" pod "3102e53e-32ab-4b1b-9341-5f6faf806cc3" (UID: "3102e53e-32ab-4b1b-9341-5f6faf806cc3"). InnerVolumeSpecName "kube-api-access-fb2vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:31.853578 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.853569    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-xks-ca" pod "3102e53e-32ab-4b1b-9341-5f6faf806cc3" (UID: "3102e53e-32ab-4b1b-9341-5f6faf806cc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:31.952623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.952591    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:31.952623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.952619    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3102e53e-32ab-4b1b-9341-5f6faf806cc3-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:31.952623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:31.952629    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb2vg\" (UniqueName: \"kubernetes.io/projected/3102e53e-32ab-4b1b-9341-5f6faf806cc3-kube-api-access-fb2vg\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:32.043317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.043230    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"]
Nov 03 10:44:32.049279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.049246    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3102e53e-32ab-4b1b-9341-5f6faf806cc3" containerName="revoke-keys"
Nov 03 10:44:32.049508 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.049492    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3102e53e-32ab-4b1b-9341-5f6faf806cc3" containerName="revoke-keys"
Nov 03 10:44:32.049784 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.049771    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3102e53e-32ab-4b1b-9341-5f6faf806cc3" containerName="revoke-keys"
Nov 03 10:44:32.071408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.070580    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187191    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187243    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187336    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187423    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187450    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187493    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187525    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187554    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.239004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.187675    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgtd\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-kube-api-access-jcgtd\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.261355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.261318    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"]
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.288681    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgtd\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-kube-api-access-jcgtd\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.288790    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.288817    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.288886    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.288981    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.289008    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.289048    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.289079    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.289106    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.289583    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.290193    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.291137    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.291396    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.303010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.292255    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.314941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.308567    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgtd\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-kube-api-access-jcgtd\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.337686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.337598    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.338582 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.338552    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.338735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.338709    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.455544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.455513    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:32.456792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.456770    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:44:32.456866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.456817    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 10:44:32.539947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.539267    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"
Nov 03 10:44:32.539947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.539361    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595" event={"ID":"3102e53e-32ab-4b1b-9341-5f6faf806cc3","Type":"ContainerDied","Data":"dbde45d906f16dd42c1ffa6b3f2b135f05c9355f2d47af7a9c5e8f1eaea4d115"}
Nov 03 10:44:32.539947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.539414    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbde45d906f16dd42c1ffa6b3f2b135f05c9355f2d47af7a9c5e8f1eaea4d115"
Nov 03 10:44:32.550651 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.550624    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:32.561205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.561174    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 10:44:32.561320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.561229    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 10:44:32.941945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.932658    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"]
Nov 03 10:44:32.945542 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:44:32.945495    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6de65f4_8d63_4a2b_b3e5_dc462e3f84d0.slice/crio-397df1d0192fa33f8dfee186fbb7950ca96821f7a00b554b4740d8504d012eb7 WatchSource:0}: Error finding container 397df1d0192fa33f8dfee186fbb7950ca96821f7a00b554b4740d8504d012eb7: Status 404 returned error can't find the container with id 397df1d0192fa33f8dfee186fbb7950ca96821f7a00b554b4740d8504d012eb7
Nov 03 10:44:32.949421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.949237    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:32.949421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.949326    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:32.949421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:32.949369    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:33.455952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:33.455903    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:44:33.456431 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:33.456407    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 10:44:33.550056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:33.550024    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" event={"ID":"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0","Type":"ContainerStarted","Data":"3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d"}
Nov 03 10:44:33.550499 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:33.550481    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" event={"ID":"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0","Type":"ContainerStarted","Data":"397df1d0192fa33f8dfee186fbb7950ca96821f7a00b554b4740d8504d012eb7"}
Nov 03 10:44:33.551096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:33.551080    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:33.561937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:33.561888    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 10:44:33.562094 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:33.561980    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 10:44:33.571213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:33.571165    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podStartSLOduration=1.571149548 podStartE2EDuration="1.571149548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:33.569610368 +0000 UTC m=+917.474464803" watchObservedRunningTime="2026-11-03 10:44:33.571149548 +0000 UTC m=+917.476003986"
Nov 03 10:44:34.455880 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:34.455838    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:44:34.456126 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:34.455907    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 10:44:34.554038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:34.553995    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:44:34.554553 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:34.554058    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 10:44:34.560943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:34.560899    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 10:44:34.561072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:34.560965    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 10:44:35.458355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:35.458320    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:44:35.458642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:35.458615    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 10:44:35.558944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:35.558445    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:44:35.558944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:35.558501    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 10:44:36.352742 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.352700    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:36.354647 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.353706    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:36.355400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.354958    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 10:44:36.465944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.462293    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:44:36.465944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.462350    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 10:44:36.564948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.562112    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:44:36.564948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.562175    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 10:44:36.648942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.647982    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:36.673034 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:36.673004    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:37.075758 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.075659    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"]
Nov 03 10:44:37.080502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.080442    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.138998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.137837    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"kube-root-ca.crt\""
Nov 03 10:44:37.138998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.138069    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"distinct-2-w4-d5d3c3-kserve-self-signed-certs\""
Nov 03 10:44:37.138998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.138344    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"default-dockercfg-j8cg6\""
Nov 03 10:44:37.138998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.138469    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d5d3c3\"/\"openshift-service-ca.crt\""
Nov 03 10:44:37.153638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.153058    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2b85\" (UniqueName: \"kubernetes.io/projected/e2977665-bb0a-4a29-a5a1-5bf515156271-kube-api-access-b2b85\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.153638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.153140    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-dshm\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.153638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.153172    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-model-cache\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.153638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.153227    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-home\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.153638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.153251    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2977665-bb0a-4a29-a5a1-5bf515156271-tls-certs\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.153638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.153328    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-tmp-dir\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.154882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.154851    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"]
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.265655    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2b85\" (UniqueName: \"kubernetes.io/projected/e2977665-bb0a-4a29-a5a1-5bf515156271-kube-api-access-b2b85\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.265813    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-dshm\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.265849    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-model-cache\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.265938    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-home\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.265966    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2977665-bb0a-4a29-a5a1-5bf515156271-tls-certs\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.266053    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-tmp-dir\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.266577    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-tmp-dir\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.268185    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-home\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.268949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.268422    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-model-cache\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.327993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.301506    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-dshm\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.329425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.329397    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2977665-bb0a-4a29-a5a1-5bf515156271-tls-certs\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.358201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.355445    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2b85\" (UniqueName: \"kubernetes.io/projected/e2977665-bb0a-4a29-a5a1-5bf515156271-kube-api-access-b2b85\") pod \"distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") " pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.456783 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.456284    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 10:44:37.456783 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.456351    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 10:44:37.458674 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.458434    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:37.551514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.551408    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:44:37.551514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.551470    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 10:44:37.836328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.836291    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:37.856811 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:37.856780    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"]
Nov 03 10:44:38.552328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:38.552287    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:44:38.552525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:38.552346    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 10:44:38.892980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:38.890125    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" event={"ID":"e2977665-bb0a-4a29-a5a1-5bf515156271","Type":"ContainerStarted","Data":"9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11"}
Nov 03 10:44:38.892980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:38.890192    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" event={"ID":"e2977665-bb0a-4a29-a5a1-5bf515156271","Type":"ContainerStarted","Data":"8746b99f3822868ef646e89f9258c2424be73c9af76d9366e1cd3bb01e02aeac"}
Nov 03 10:44:38.937865 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:38.937803    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podStartSLOduration=1.9377871629999999 podStartE2EDuration="1.937787163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:38.930428889 +0000 UTC m=+922.835283331" watchObservedRunningTime="2026-11-03 10:44:38.937787163 +0000 UTC m=+922.842641604"
Nov 03 10:44:39.214203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:39.214074    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:39.215148 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:39.215122    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:39.217174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:39.217148    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 10:44:39.459106 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:39.459072    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:39.551335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:39.551250    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 10:44:39.551335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:39.551306    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 10:44:40.699986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:40.699954    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:40.700590 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:40.700258    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:40.701200 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:40.701181    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 10:44:42.140120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.140040    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.391133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.391026    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.618078 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.618028    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.639947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.639647    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"]
Nov 03 10:44:42.652884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.652786    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.680957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.678817    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"]
Nov 03 10:44:42.710955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.705092    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.732945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.731667    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760345    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760423    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760454    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760489    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760516    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760589    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760634    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxg7\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-kube-api-access-mgxg7\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760668    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.765948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.760697    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.871975 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.871911    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxg7\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-kube-api-access-mgxg7\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.899326 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.899285    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.899636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.899591    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.900031 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.900010    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.900269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.900251    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.900422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.900403    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.927023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.926899    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.927304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.927260    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.927806 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.927780    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.929792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.928018    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.930007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.928321    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.945461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.928527    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.945789 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.928721    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.945981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.928996    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:42.964985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.951122    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"]
Nov 03 10:44:42.964985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.951175    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"]
Nov 03 10:44:42.964985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:42.951276    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.058251 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.058209    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded"
Nov 03 10:44:43.071951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.059127    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59dsn\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-kube-api-access-59dsn\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.130073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.130033    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.131126 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.131098    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.136175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.136147    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.161080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.161036    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.173947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.161737    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.174240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.174214    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.174362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.174347    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da57fce0-41d2-4546-a567-7dcc3bdc9221-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.183181 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.183092    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.183517 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.079747    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxg7\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-kube-api-access-mgxg7\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:43.183663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.084565    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:43.183771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.084845    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:43.183872 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.098530    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.343665    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.343734    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.343779    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.343824    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.343857    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.343891    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da57fce0-41d2-4546-a567-7dcc3bdc9221-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.343991    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.344086    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59dsn\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-kube-api-access-59dsn\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.344122    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.356186    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.357167    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da57fce0-41d2-4546-a567-7dcc3bdc9221-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.357471    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.357713    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.362942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.358045    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.369171 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.369139    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:43.383953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.371458    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.407949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.393372    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:44:43.407949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.394024    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:43.407949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.397596    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59dsn\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-kube-api-access-59dsn\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.460989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.457952    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.460989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.459183    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.475985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.461302    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.477401 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.477182    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:43.641234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.641194    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:44:43.670057 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.656739    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.728182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.724020    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:44:43.747135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.737020    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:43.747135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.737338    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 10:44:43.747135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:43.737558    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:44:44.171946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.168029    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.251962 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:44:44.221232    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:44:44.402732 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.402091    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:44:44.498544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.497136    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.499252 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.499224    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:44:44.686001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.624451    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.714093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.710471    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.714093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.710801    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:44:44.728035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.724609    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:44.753165 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.753132    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 10:44:44.757117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.757076    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:44.843592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:44.753039    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 10:44:45.176980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.176647    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.204942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.204368    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:44:45.415630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.415580    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"]
Nov 03 10:44:45.416024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.415985    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.456050 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.454015    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.576301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.540016    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:44:45.576301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.541436    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"]
Nov 03 10:44:45.659058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.630030    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.659058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.630733    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:44:45.731719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.731678    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.737695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.737658    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.773786 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.756968    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:45.891944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:45.864223    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.102184 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.098278    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.233992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.213000    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.289950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.282107    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.330001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.217725    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.510164 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.510100    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.559392 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.161039    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" event={"ID":"ea18e571-df1d-4711-aa0c-1c11ca6be4c3","Type":"ContainerStarted","Data":"288c2023a0bb5db13dfe2ee16d778186bbfdf720dbd31bd28d4d51159b328dd8"}
Nov 03 10:44:46.560015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.178345    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 03 10:44:46.560472 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.218390    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:44:46.560880 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.550590    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.561143 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.550629    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:44:46.750946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.723193    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:44:46.750946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.748421    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.754037 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.753672    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.783533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.783389    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.801842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.798062    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:44:46.825208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.825113    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:44:46.907422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:46.906023    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 10:44:47.266452 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.264702    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" event={"ID":"ea18e571-df1d-4711-aa0c-1c11ca6be4c3","Type":"ContainerStarted","Data":"dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20"}
Nov 03 10:44:47.270708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.270671    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" event={"ID":"da57fce0-41d2-4546-a567-7dcc3bdc9221","Type":"ContainerStarted","Data":"45dfa602b1b3e394786c01746eae822ac9b4b82866d7c2c291e15263c22e9043"}
Nov 03 10:44:47.310114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.310073    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:44:47.358726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.348262    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podStartSLOduration=5.348239378 podStartE2EDuration="5.348239378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:47.307660343 +0000 UTC m=+931.212514785" watchObservedRunningTime="2026-11-03 10:44:47.348239378 +0000 UTC m=+931.253093818"
Nov 03 10:44:47.363501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.362778    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:44:47.374003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.373974    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:47.374889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.374855    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 10:44:47.375034 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.374964    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 10:44:47.379944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.378544    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:44:47.409965 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.405172    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:44:47.410150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.410022    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:44:47.428142 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.428110    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:44:47.472585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.472559    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:47.498733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.495026    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:44:47.498733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.497475    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:44:47.518591 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:47.518563    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:48.278335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:48.278303    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" event={"ID":"da57fce0-41d2-4546-a567-7dcc3bdc9221","Type":"ContainerStarted","Data":"041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8"}
Nov 03 10:44:48.333134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:48.333067    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podStartSLOduration=6.333040224 podStartE2EDuration="6.333040224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:44:48.307686166 +0000 UTC m=+932.212540609" watchObservedRunningTime="2026-11-03 10:44:48.333040224 +0000 UTC m=+932.237894665"
Nov 03 10:44:48.347109 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:48.347082    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:44:48.371994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:48.371963    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 10:44:48.372160 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:48.372019    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 10:44:48.500596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:48.497728    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:48.513125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:48.506092    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 10:44:48.513125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:48.506162    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 10:44:49.372993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:49.372352    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 10:44:49.372993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:49.372406    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 10:44:49.508954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:49.492558    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 10:44:49.508954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:49.492632    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 10:44:50.372213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:50.372174    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 10:44:50.372415 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:50.372226    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 10:44:50.498955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:50.495421    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 10:44:50.498955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:50.495498    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 10:44:51.379941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:51.374429    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 10:44:51.379941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:51.374520    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 10:44:51.495781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:51.495316    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 10:44:51.495781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:51.495378    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 10:44:52.373424 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:52.373388    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 10:44:52.374071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:52.374036    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 10:44:52.494338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:52.494299    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 10:44:52.494581 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:52.494552    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 10:44:53.375946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:53.373101    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 10:44:53.375946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:53.373177    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 10:44:53.492551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:53.492516    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 10:44:53.492840 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:53.492811    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 10:44:54.372361 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.372319    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 10:44:54.372592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.372403    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 10:44:54.817200 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.817105    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"]
Nov 03 10:44:54.822873 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.822759    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:54.826673 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.826642    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:44:54.827691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.827665    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:44:54.828861 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.828835    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"]
Nov 03 10:44:54.932531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.932059    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:54.932771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.932753    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:54.936758 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.936738    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 10:44:54.950898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.950866    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:54.950898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.950898    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njcsr\" (UniqueName: \"kubernetes.io/projected/9979c146-a608-4130-9ad4-ad216056b5a9-kube-api-access-njcsr\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:54.951106 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:54.950983    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:55.052509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.052477    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:55.052666 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.052650    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:55.052726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.052679    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcsr\" (UniqueName: \"kubernetes.io/projected/9979c146-a608-4130-9ad4-ad216056b5a9-kube-api-access-njcsr\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:55.053751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.053726    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:55.056010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.055986    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:55.062332 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.062308    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcsr\" (UniqueName: \"kubernetes.io/projected/9979c146-a608-4130-9ad4-ad216056b5a9-kube-api-access-njcsr\") pod \"maas-api-revoke-sub-8c6ebd9d-khmg6\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:55.151482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.151452    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:55.378236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.378205    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:55.378545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.378518    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:55.379290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.379273    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 10:44:55.458911 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:55.457816    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"]
Nov 03 10:44:55.461138 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:44:55.461114    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9979c146_a608_4130_9ad4_ad216056b5a9.slice/crio-941f04e3a25e9c2aaf86ada248b2f729845b75715be085f9818121fbc0bb2ac2 WatchSource:0}: Error finding container 941f04e3a25e9c2aaf86ada248b2f729845b75715be085f9818121fbc0bb2ac2: Status 404 returned error can't find the container with id 941f04e3a25e9c2aaf86ada248b2f729845b75715be085f9818121fbc0bb2ac2
Nov 03 10:44:56.463722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:56.463690    5393 generic.go:356] "Generic (PLEG): container finished" podID="9979c146-a608-4130-9ad4-ad216056b5a9" containerID="8424ee67cefbf50c5ff6ab0e04bfe55aa79527ec7c7e0af06d726e6cee657d4f" exitCode=0
Nov 03 10:44:56.464116 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:56.463784    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6" event={"ID":"9979c146-a608-4130-9ad4-ad216056b5a9","Type":"ContainerDied","Data":"8424ee67cefbf50c5ff6ab0e04bfe55aa79527ec7c7e0af06d726e6cee657d4f"}
Nov 03 10:44:56.464116 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:56.463846    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6" event={"ID":"9979c146-a608-4130-9ad4-ad216056b5a9","Type":"ContainerStarted","Data":"941f04e3a25e9c2aaf86ada248b2f729845b75715be085f9818121fbc0bb2ac2"}
Nov 03 10:44:57.750461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:57.750434    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:57.923066 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:57.923029    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9979c146-a608-4130-9ad4-ad216056b5a9-kube-api-access-njcsr\" (UniqueName: \"kubernetes.io/projected/9979c146-a608-4130-9ad4-ad216056b5a9-kube-api-access-njcsr\") pod \"9979c146-a608-4130-9ad4-ad216056b5a9\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") "
Nov 03 10:44:57.923262 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:57.923081    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-openshift-ca\") pod \"9979c146-a608-4130-9ad4-ad216056b5a9\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") "
Nov 03 10:44:57.923262 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:57.923172    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-xks-ca\") pod \"9979c146-a608-4130-9ad4-ad216056b5a9\" (UID: \"9979c146-a608-4130-9ad4-ad216056b5a9\") "
Nov 03 10:44:57.923428 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:57.923399    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-openshift-ca" pod "9979c146-a608-4130-9ad4-ad216056b5a9" (UID: "9979c146-a608-4130-9ad4-ad216056b5a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:44:57.923702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:57.923682    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:57.925353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:57.925326    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-xks-ca" pod "9979c146-a608-4130-9ad4-ad216056b5a9" (UID: "9979c146-a608-4130-9ad4-ad216056b5a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:44:57.925504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:57.925483    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9979c146-a608-4130-9ad4-ad216056b5a9-kube-api-access-njcsr" pod "9979c146-a608-4130-9ad4-ad216056b5a9" (UID: "9979c146-a608-4130-9ad4-ad216056b5a9"). InnerVolumeSpecName "kube-api-access-njcsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:44:58.025239 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:58.025139    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njcsr\" (UniqueName: \"kubernetes.io/projected/9979c146-a608-4130-9ad4-ad216056b5a9-kube-api-access-njcsr\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:58.025239 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:58.025183    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979c146-a608-4130-9ad4-ad216056b5a9-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:44:58.487292 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:58.487260    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"
Nov 03 10:44:58.487514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:58.487301    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6" event={"ID":"9979c146-a608-4130-9ad4-ad216056b5a9","Type":"ContainerDied","Data":"941f04e3a25e9c2aaf86ada248b2f729845b75715be085f9818121fbc0bb2ac2"}
Nov 03 10:44:58.487514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:44:58.487360    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941f04e3a25e9c2aaf86ada248b2f729845b75715be085f9818121fbc0bb2ac2"
Nov 03 10:45:00.180534 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.180500    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"]
Nov 03 10:45:00.182343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.182314    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9979c146-a608-4130-9ad4-ad216056b5a9" containerName="revoke-keys"
Nov 03 10:45:00.182733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.182709    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9979c146-a608-4130-9ad4-ad216056b5a9" containerName="revoke-keys"
Nov 03 10:45:00.183152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.183134    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9979c146-a608-4130-9ad4-ad216056b5a9" containerName="revoke-keys"
Nov 03 10:45:00.203509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.203480    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:00.229506 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.229477    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:45:00.352961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.326389    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0454f4d8-c98f-4eb2-84b1-cb55441c6709-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895045-57bxv\" (UID: \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:00.352961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.326452    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46c8r\" (UniqueName: \"kubernetes.io/projected/0454f4d8-c98f-4eb2-84b1-cb55441c6709-kube-api-access-46c8r\") pod \"maas-api-key-cleanup-29895045-57bxv\" (UID: \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:00.359032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.358991    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"]
Nov 03 10:45:00.367009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.366969    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:00.428973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.428941    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7\" (UID: \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:00.429242 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.429220    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsq4\" (UniqueName: \"kubernetes.io/projected/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-kube-api-access-7jsq4\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7\" (UID: \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:00.429569 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.429432    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0454f4d8-c98f-4eb2-84b1-cb55441c6709-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895045-57bxv\" (UID: \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:00.429797 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.429779    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46c8r\" (UniqueName: \"kubernetes.io/projected/0454f4d8-c98f-4eb2-84b1-cb55441c6709-kube-api-access-46c8r\") pod \"maas-api-key-cleanup-29895045-57bxv\" (UID: \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:00.431093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.431043    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0454f4d8-c98f-4eb2-84b1-cb55441c6709-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895045-57bxv\" (UID: \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:00.446772 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.446736    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"]
Nov 03 10:45:00.464410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.464381    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:00.468152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.468120    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46c8r\" (UniqueName: \"kubernetes.io/projected/0454f4d8-c98f-4eb2-84b1-cb55441c6709-kube-api-access-46c8r\") pod \"maas-api-key-cleanup-29895045-57bxv\" (UID: \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:00.474362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.474287    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"]
Nov 03 10:45:00.482288 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.479433    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"]
Nov 03 10:45:00.489759 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.489681    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"]
Nov 03 10:45:00.535943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.535272    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842\" (UID: \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:00.558939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.555142    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7\" (UID: \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:00.558939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.555293    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsq4\" (UniqueName: \"kubernetes.io/projected/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-kube-api-access-7jsq4\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7\" (UID: \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:00.604952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.604395    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmmn\" (UniqueName: \"kubernetes.io/projected/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-kube-api-access-4tmmn\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842\" (UID: \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:00.627948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.606208    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7\" (UID: \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:00.660963 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.650885    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsq4\" (UniqueName: \"kubernetes.io/projected/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-kube-api-access-7jsq4\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7\" (UID: \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:00.714605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.712806    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842\" (UID: \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:00.714605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.713070    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmmn\" (UniqueName: \"kubernetes.io/projected/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-kube-api-access-4tmmn\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842\" (UID: \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:00.714605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.714486    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842\" (UID: \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:00.758946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.745475    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:00.758946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.746491    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmmn\" (UniqueName: \"kubernetes.io/projected/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-kube-api-access-4tmmn\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842\" (UID: \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:00.829951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.817714    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:00.829951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:00.820633    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:01.547185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:01.546005    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:45:01.598946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:01.596024    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.752949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:01.749980    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.763060 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:01.763012    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.824309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:01.823021    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:01.852206 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:01.851019    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.075164 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:02.075054    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.334009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:02.305123    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.570469 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:02.570417    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.787357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:02.746530    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:45:02.787357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:02.770697    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:45:02.904556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:02.844608    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:02.904556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:02.844849    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.030589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:02.896899    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.056529 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.056431    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.143629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.108247    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"]
Nov 03 10:45:03.222215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.221095    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:45:03.222215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.221145    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:45:03.222215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.221179    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:45:03.222215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.221214    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:45:03.222215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.221244    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:45:03.222215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.221272    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:45:03.368803 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.366011    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.368803 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.366692    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:45:03.455948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.421348    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:45:03.479986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.456741    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.757981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.750266    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:45:03.757981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.751331    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.791455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.791413    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.884577 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.868008    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:03.887228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:03.887193    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 10:45:04.093072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.048653    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.093072 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:45:04.081602    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3797e5_c99a_4c21_bfd3_45a70ee032bd.slice/crio-017bb0455c96fc43109c975e43317879a9727b4e449a57569bcdfa007837c099 WatchSource:0}: Error finding container 017bb0455c96fc43109c975e43317879a9727b4e449a57569bcdfa007837c099: Status 404 returned error can't find the container with id 017bb0455c96fc43109c975e43317879a9727b4e449a57569bcdfa007837c099
Nov 03 10:45:04.201660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.201021    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:45:04.485298 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.446775    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" event={"ID":"3d4cc4b8-ef31-416f-952a-7c4cfc54c411","Type":"ContainerStarted","Data":"5c9971d4a7139b2b033c13722070f4f5fdc3c9b44b961a2312532ed540287a18"}
Nov 03 10:45:04.485298 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.449472    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.759636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.759590    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.762541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.761584    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"]
Nov 03 10:45:04.762893 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.762873    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:45:04.766193 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.766167    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.767351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.767324    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.767855 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.767836    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:45:04.869173 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.869131    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:04.887952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:04.887705    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.065698 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.064199    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:05.075125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.072881    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"]
Nov 03 10:45:05.138699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.137319    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:45:05.138699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.137854    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:45:05.140740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.140711    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:45:05.147593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.147568    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:45:05.169834 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.162118    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:45:05.169834 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.163526    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:45:05.171660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.170973    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:45:05.289078 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.284808    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"]
Nov 03 10:45:05.300886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.299136    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.307475 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.307451    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"]
Nov 03 10:45:05.308042 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.308018    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:05.418374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.414154    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.421625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.420016    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrh4\" (UniqueName: \"kubernetes.io/projected/1bed52ed-e881-4545-8aac-5c17bcee11ef-kube-api-access-bsrh4\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.421625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.420143    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.466177 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.466138    5393 generic.go:356] "Generic (PLEG): container finished" podID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerID="c2e4770a4001d7b2b50874aa5f8b105c27ef931005a312826f72dc6faecf4811" exitCode=60
Nov 03 10:45:05.466380 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.466228    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" event={"ID":"3d4cc4b8-ef31-416f-952a-7c4cfc54c411","Type":"ContainerDied","Data":"c2e4770a4001d7b2b50874aa5f8b105c27ef931005a312826f72dc6faecf4811"}
Nov 03 10:45:05.466662 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.466638    5393 scope.go:122] "RemoveContainer" containerID="c2e4770a4001d7b2b50874aa5f8b105c27ef931005a312826f72dc6faecf4811"
Nov 03 10:45:05.468944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.468628    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" event={"ID":"0454f4d8-c98f-4eb2-84b1-cb55441c6709","Type":"ContainerStarted","Data":"03ef4df82908a1c10c5210dfa83e024ba10f182e47e737b551822e4378c05e5f"}
Nov 03 10:45:05.472372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.472342    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" event={"ID":"ad3797e5-c99a-4c21-bfd3-45a70ee032bd","Type":"ContainerStarted","Data":"fd45b5ad1b8616f06f40a3900d9d60ec5b35d19ec97d940ed9b46879ef8a768e"}
Nov 03 10:45:05.472474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.472399    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" event={"ID":"ad3797e5-c99a-4c21-bfd3-45a70ee032bd","Type":"ContainerStarted","Data":"017bb0455c96fc43109c975e43317879a9727b4e449a57569bcdfa007837c099"}
Nov 03 10:45:05.499863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.499806    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" podStartSLOduration=5.499788529 podStartE2EDuration="5.499788529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:05.497431622 +0000 UTC m=+949.402286058" watchObservedRunningTime="2026-11-03 10:45:05.499788529 +0000 UTC m=+949.404642972"
Nov 03 10:45:05.521004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.520971    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrh4\" (UniqueName: \"kubernetes.io/projected/1bed52ed-e881-4545-8aac-5c17bcee11ef-kube-api-access-bsrh4\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.521152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.521076    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.521227 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.521197    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.522253 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.522225    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.524778 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.524720    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.532241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.531348    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrh4\" (UniqueName: \"kubernetes.io/projected/1bed52ed-e881-4545-8aac-5c17bcee11ef-kube-api-access-bsrh4\") pod \"maas-api-revoke-sub-2d7d2989-glbdx\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.660209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.656132    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:05.956876 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:05.956852    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"]
Nov 03 10:45:05.958842 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:45:05.958809    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bed52ed_e881_4545_8aac_5c17bcee11ef.slice/crio-bdf4120a15e6d9dce9c9055c72fd377bd2631ec33c9ee82d108740996141ad03 WatchSource:0}: Error finding container bdf4120a15e6d9dce9c9055c72fd377bd2631ec33c9ee82d108740996141ad03: Status 404 returned error can't find the container with id bdf4120a15e6d9dce9c9055c72fd377bd2631ec33c9ee82d108740996141ad03
Nov 03 10:45:06.480182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.480105    5393 generic.go:356] "Generic (PLEG): container finished" podID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerID="fd45b5ad1b8616f06f40a3900d9d60ec5b35d19ec97d940ed9b46879ef8a768e" exitCode=60
Nov 03 10:45:06.480762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.480184    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" event={"ID":"ad3797e5-c99a-4c21-bfd3-45a70ee032bd","Type":"ContainerDied","Data":"fd45b5ad1b8616f06f40a3900d9d60ec5b35d19ec97d940ed9b46879ef8a768e"}
Nov 03 10:45:06.480762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.480476    5393 scope.go:122] "RemoveContainer" containerID="fd45b5ad1b8616f06f40a3900d9d60ec5b35d19ec97d940ed9b46879ef8a768e"
Nov 03 10:45:06.482215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.482184    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx" event={"ID":"1bed52ed-e881-4545-8aac-5c17bcee11ef","Type":"ContainerStarted","Data":"3297bf5f54f38425119df1bf8dba0148484e4aefed5a9a227354514b8e8ac2af"}
Nov 03 10:45:06.482319 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.482254    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx" event={"ID":"1bed52ed-e881-4545-8aac-5c17bcee11ef","Type":"ContainerStarted","Data":"bdf4120a15e6d9dce9c9055c72fd377bd2631ec33c9ee82d108740996141ad03"}
Nov 03 10:45:06.486478 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.486455    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" event={"ID":"3d4cc4b8-ef31-416f-952a-7c4cfc54c411","Type":"ContainerStarted","Data":"e46a9f33fdb280bb3b1c8f701cf7d8abb9ead7aca5e0fef17ff53a63fa26f653"}
Nov 03 10:45:06.488390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.488365    5393 generic.go:356] "Generic (PLEG): container finished" podID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerID="710064fc32d61d40da1a32230dcaa3c802f68326a988033e5ab9ef13eab67567" exitCode=60
Nov 03 10:45:06.488489 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.488432    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" event={"ID":"0454f4d8-c98f-4eb2-84b1-cb55441c6709","Type":"ContainerDied","Data":"710064fc32d61d40da1a32230dcaa3c802f68326a988033e5ab9ef13eab67567"}
Nov 03 10:45:06.488681 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.488665    5393 scope.go:122] "RemoveContainer" containerID="710064fc32d61d40da1a32230dcaa3c802f68326a988033e5ab9ef13eab67567"
Nov 03 10:45:06.586273 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.586194    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx" podStartSLOduration=1.586172962 podStartE2EDuration="1.586172962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:06.559262098 +0000 UTC m=+950.464116541" watchObservedRunningTime="2026-11-03 10:45:06.586172962 +0000 UTC m=+950.491027405"
Nov 03 10:45:06.604366 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:06.604299    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" podStartSLOduration=6.604284303 podStartE2EDuration="6.604284303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:06.595287913 +0000 UTC m=+950.500142355" watchObservedRunningTime="2026-11-03 10:45:06.604284303 +0000 UTC m=+950.509138744"
Nov 03 10:45:07.499289 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.499253    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" event={"ID":"ad3797e5-c99a-4c21-bfd3-45a70ee032bd","Type":"ContainerStarted","Data":"108b65b83a353660d0d00a148e322241fc123d4e9d2745d6a4cb9be473a4316c"}
Nov 03 10:45:07.499733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.499637    5393 scope.go:122] "RemoveContainer" containerID="108b65b83a353660d0d00a148e322241fc123d4e9d2745d6a4cb9be473a4316c"
Nov 03 10:45:07.500047 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:07.500013    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7_odh-ai-gateway-infra(ad3797e5-c99a-4c21-bfd3-45a70ee032bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd"
Nov 03 10:45:07.502228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.502194    5393 generic.go:356] "Generic (PLEG): container finished" podID="1bed52ed-e881-4545-8aac-5c17bcee11ef" containerID="3297bf5f54f38425119df1bf8dba0148484e4aefed5a9a227354514b8e8ac2af" exitCode=0
Nov 03 10:45:07.502337 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.502293    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx" event={"ID":"1bed52ed-e881-4545-8aac-5c17bcee11ef","Type":"ContainerDied","Data":"3297bf5f54f38425119df1bf8dba0148484e4aefed5a9a227354514b8e8ac2af"}
Nov 03 10:45:07.504503 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.504482    5393 generic.go:356] "Generic (PLEG): container finished" podID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerID="e46a9f33fdb280bb3b1c8f701cf7d8abb9ead7aca5e0fef17ff53a63fa26f653" exitCode=60
Nov 03 10:45:07.504610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.504552    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" event={"ID":"3d4cc4b8-ef31-416f-952a-7c4cfc54c411","Type":"ContainerDied","Data":"e46a9f33fdb280bb3b1c8f701cf7d8abb9ead7aca5e0fef17ff53a63fa26f653"}
Nov 03 10:45:07.504610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.504597    5393 scope.go:122] "RemoveContainer" containerID="c2e4770a4001d7b2b50874aa5f8b105c27ef931005a312826f72dc6faecf4811"
Nov 03 10:45:07.504843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.504823    5393 scope.go:122] "RemoveContainer" containerID="e46a9f33fdb280bb3b1c8f701cf7d8abb9ead7aca5e0fef17ff53a63fa26f653"
Nov 03 10:45:07.505290 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:07.505266    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842_odh-ai-gateway-infra(3d4cc4b8-ef31-416f-952a-7c4cfc54c411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411"
Nov 03 10:45:07.506844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.506826    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" event={"ID":"0454f4d8-c98f-4eb2-84b1-cb55441c6709","Type":"ContainerStarted","Data":"3cc00b82d093955624e89bfe04dd5391e57a1341a8eb88fe90c86e5c3a46b0fc"}
Nov 03 10:45:07.532315 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:07.532261    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" podStartSLOduration=7.532247637 podStartE2EDuration="7.532247637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:07.531046524 +0000 UTC m=+951.435900960" watchObservedRunningTime="2026-11-03 10:45:07.532247637 +0000 UTC m=+951.437102077"
Nov 03 10:45:08.516269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.516234    5393 scope.go:122] "RemoveContainer" containerID="e46a9f33fdb280bb3b1c8f701cf7d8abb9ead7aca5e0fef17ff53a63fa26f653"
Nov 03 10:45:08.516727 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:08.516606    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842_odh-ai-gateway-infra(3d4cc4b8-ef31-416f-952a-7c4cfc54c411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411"
Nov 03 10:45:08.518144 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.518106    5393 generic.go:356] "Generic (PLEG): container finished" podID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerID="3cc00b82d093955624e89bfe04dd5391e57a1341a8eb88fe90c86e5c3a46b0fc" exitCode=60
Nov 03 10:45:08.518241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.518193    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" event={"ID":"0454f4d8-c98f-4eb2-84b1-cb55441c6709","Type":"ContainerDied","Data":"3cc00b82d093955624e89bfe04dd5391e57a1341a8eb88fe90c86e5c3a46b0fc"}
Nov 03 10:45:08.518289 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.518253    5393 scope.go:122] "RemoveContainer" containerID="710064fc32d61d40da1a32230dcaa3c802f68326a988033e5ab9ef13eab67567"
Nov 03 10:45:08.518509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.518485    5393 scope.go:122] "RemoveContainer" containerID="3cc00b82d093955624e89bfe04dd5391e57a1341a8eb88fe90c86e5c3a46b0fc"
Nov 03 10:45:08.518912 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:08.518864    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895045-57bxv_odh-ai-gateway-infra(0454f4d8-c98f-4eb2-84b1-cb55441c6709)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709"
Nov 03 10:45:08.520761 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.520738    5393 generic.go:356] "Generic (PLEG): container finished" podID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerID="108b65b83a353660d0d00a148e322241fc123d4e9d2745d6a4cb9be473a4316c" exitCode=60
Nov 03 10:45:08.520860 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.520789    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" event={"ID":"ad3797e5-c99a-4c21-bfd3-45a70ee032bd","Type":"ContainerDied","Data":"108b65b83a353660d0d00a148e322241fc123d4e9d2745d6a4cb9be473a4316c"}
Nov 03 10:45:08.521158 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.521139    5393 scope.go:122] "RemoveContainer" containerID="108b65b83a353660d0d00a148e322241fc123d4e9d2745d6a4cb9be473a4316c"
Nov 03 10:45:08.521414 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:08.521395    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7_odh-ai-gateway-infra(ad3797e5-c99a-4c21-bfd3-45a70ee032bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd"
Nov 03 10:45:08.538217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.532053    5393 scope.go:122] "RemoveContainer" containerID="fd45b5ad1b8616f06f40a3900d9d60ec5b35d19ec97d940ed9b46879ef8a768e"
Nov 03 10:45:08.685980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.685954    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:08.792017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.791933    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-xks-ca\") pod \"1bed52ed-e881-4545-8aac-5c17bcee11ef\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") "
Nov 03 10:45:08.792017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.792012    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bed52ed-e881-4545-8aac-5c17bcee11ef-kube-api-access-bsrh4\" (UniqueName: \"kubernetes.io/projected/1bed52ed-e881-4545-8aac-5c17bcee11ef-kube-api-access-bsrh4\") pod \"1bed52ed-e881-4545-8aac-5c17bcee11ef\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") "
Nov 03 10:45:08.792218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.792205    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-openshift-ca\") pod \"1bed52ed-e881-4545-8aac-5c17bcee11ef\" (UID: \"1bed52ed-e881-4545-8aac-5c17bcee11ef\") "
Nov 03 10:45:08.792572 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.792543    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-openshift-ca" pod "1bed52ed-e881-4545-8aac-5c17bcee11ef" (UID: "1bed52ed-e881-4545-8aac-5c17bcee11ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:08.794290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.794266    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bed52ed-e881-4545-8aac-5c17bcee11ef-kube-api-access-bsrh4" pod "1bed52ed-e881-4545-8aac-5c17bcee11ef" (UID: "1bed52ed-e881-4545-8aac-5c17bcee11ef"). InnerVolumeSpecName "kube-api-access-bsrh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:08.794500 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.794481    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-xks-ca" pod "1bed52ed-e881-4545-8aac-5c17bcee11ef" (UID: "1bed52ed-e881-4545-8aac-5c17bcee11ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:08.894146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.894060    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:08.894146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.894097    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsrh4\" (UniqueName: \"kubernetes.io/projected/1bed52ed-e881-4545-8aac-5c17bcee11ef-kube-api-access-bsrh4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:08.894146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:08.894113    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bed52ed-e881-4545-8aac-5c17bcee11ef-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:09.529283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:09.529256    5393 scope.go:122] "RemoveContainer" containerID="108b65b83a353660d0d00a148e322241fc123d4e9d2745d6a4cb9be473a4316c"
Nov 03 10:45:09.529747 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:09.529558    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7_odh-ai-gateway-infra(ad3797e5-c99a-4c21-bfd3-45a70ee032bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd"
Nov 03 10:45:09.530318 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:09.530296    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"
Nov 03 10:45:09.530437 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:09.530325    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx" event={"ID":"1bed52ed-e881-4545-8aac-5c17bcee11ef","Type":"ContainerDied","Data":"bdf4120a15e6d9dce9c9055c72fd377bd2631ec33c9ee82d108740996141ad03"}
Nov 03 10:45:09.530437 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:09.530365    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf4120a15e6d9dce9c9055c72fd377bd2631ec33c9ee82d108740996141ad03"
Nov 03 10:45:09.532323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:09.532306    5393 scope.go:122] "RemoveContainer" containerID="3cc00b82d093955624e89bfe04dd5391e57a1341a8eb88fe90c86e5c3a46b0fc"
Nov 03 10:45:09.532545 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:09.532529    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895045-57bxv_odh-ai-gateway-infra(0454f4d8-c98f-4eb2-84b1-cb55441c6709)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709"
Nov 03 10:45:10.671761 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.671722    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"]
Nov 03 10:45:10.672341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.672314    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1bed52ed-e881-4545-8aac-5c17bcee11ef" containerName="revoke-keys"
Nov 03 10:45:10.672388 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.672341    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bed52ed-e881-4545-8aac-5c17bcee11ef" containerName="revoke-keys"
Nov 03 10:45:10.672439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.672430    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bed52ed-e881-4545-8aac-5c17bcee11ef" containerName="revoke-keys"
Nov 03 10:45:10.676212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.676192    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.679815 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.679790    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:10.683093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.683067    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"]
Nov 03 10:45:10.827034 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.826996    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trjd\" (UniqueName: \"kubernetes.io/projected/6f56a720-bdac-485d-ae61-f983f408a10a-kube-api-access-7trjd\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.827214 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.827154    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.827214 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.827193    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.928466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.928386    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trjd\" (UniqueName: \"kubernetes.io/projected/6f56a720-bdac-485d-ae61-f983f408a10a-kube-api-access-7trjd\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.928603 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.928476    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.928603 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.928504    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.929193 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.929165    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.931191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.931170    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.936754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.936733    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trjd\" (UniqueName: \"kubernetes.io/projected/6f56a720-bdac-485d-ae61-f983f408a10a-kube-api-access-7trjd\") pod \"maas-api-revoke-sub-c188f69f-qxl8r\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:10.992571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:10.992540    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:11.282139 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:11.282103    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"]
Nov 03 10:45:11.283601 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:45:11.283575    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f56a720_bdac_485d_ae61_f983f408a10a.slice/crio-83208f439f9692a25e3d863059f0953c8520eefc8caba01feeb284cc755b9b87 WatchSource:0}: Error finding container 83208f439f9692a25e3d863059f0953c8520eefc8caba01feeb284cc755b9b87: Status 404 returned error can't find the container with id 83208f439f9692a25e3d863059f0953c8520eefc8caba01feeb284cc755b9b87
Nov 03 10:45:11.547077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:11.546987    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r" event={"ID":"6f56a720-bdac-485d-ae61-f983f408a10a","Type":"ContainerStarted","Data":"83208f439f9692a25e3d863059f0953c8520eefc8caba01feeb284cc755b9b87"}
Nov 03 10:45:12.417743 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.417705    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"]
Nov 03 10:45:12.422857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.422827    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.445855 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.445230    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"]
Nov 03 10:45:12.555283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.555252    5393 generic.go:356] "Generic (PLEG): container finished" podID="6f56a720-bdac-485d-ae61-f983f408a10a" containerID="62b838f765e65c988087aeb8dc5fd653710bc798bcb7ebcd961665897f6caabb" exitCode=0
Nov 03 10:45:12.555462 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.555344    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r" event={"ID":"6f56a720-bdac-485d-ae61-f983f408a10a","Type":"ContainerDied","Data":"62b838f765e65c988087aeb8dc5fd653710bc798bcb7ebcd961665897f6caabb"}
Nov 03 10:45:12.557930 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.557891    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe9672-f20f-4add-965c-e93aa9492094-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.558087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.558000    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.558087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.558038    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.558184 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.558125    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.558184 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.558179    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.558269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.558253    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe9672-f20f-4add-965c-e93aa9492094-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.558344 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.558329    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.558487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.558466    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7tp8\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-kube-api-access-p7tp8\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.558555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.558547    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.659711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.659663    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7tp8\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-kube-api-access-p7tp8\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.659894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.659749    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.659894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.659779    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe9672-f20f-4add-965c-e93aa9492094-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.659894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.659805    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.659894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.659824    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660140 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660052    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660140 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660119    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660238 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660178    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe9672-f20f-4add-965c-e93aa9492094-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660238 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660208    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660274    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660348    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-credential-socket\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660490    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-data\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660593    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-certs\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.660615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.660622    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe9672-f20f-4add-965c-e93aa9492094-istiod-ca-cert\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.662344 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.662321    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-envoy\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.662692 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.662671    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe9672-f20f-4add-965c-e93aa9492094-istio-podinfo\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.667305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.667282    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-istio-token\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.667534 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.667518    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7tp8\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-kube-api-access-p7tp8\") pod \"e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") " pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:12.749008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:12.748901    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:13.053084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.048098    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"]
Nov 03 10:45:13.059832 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.059794    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.059994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.059877    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.059994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.059966    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:13.191507 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:13.191470    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:45:13.563280 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.563245    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" event={"ID":"69fe9672-f20f-4add-965c-e93aa9492094","Type":"ContainerStarted","Data":"7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14"}
Nov 03 10:45:13.563762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.563306    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" event={"ID":"69fe9672-f20f-4add-965c-e93aa9492094","Type":"ContainerStarted","Data":"7e54b05cf58fd7877a1bc4b87faa77e94ee5903907622ffd9c35261aeab8b3ea"}
Nov 03 10:45:13.585215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.585166    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podStartSLOduration=1.5851513449999999 podStartE2EDuration="1.585151345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:13.584557712 +0000 UTC m=+957.489412156" watchObservedRunningTime="2026-11-03 10:45:13.585151345 +0000 UTC m=+957.490005785"
Nov 03 10:45:13.752689 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.752035    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:13.779942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.778409    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:13.790945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.782840    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-xks-ca\") pod \"6f56a720-bdac-485d-ae61-f983f408a10a\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") "
Nov 03 10:45:13.790945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.782944    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f56a720-bdac-485d-ae61-f983f408a10a-kube-api-access-7trjd\" (UniqueName: \"kubernetes.io/projected/6f56a720-bdac-485d-ae61-f983f408a10a-kube-api-access-7trjd\") pod \"6f56a720-bdac-485d-ae61-f983f408a10a\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") "
Nov 03 10:45:13.790945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.783011    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-openshift-ca\") pod \"6f56a720-bdac-485d-ae61-f983f408a10a\" (UID: \"6f56a720-bdac-485d-ae61-f983f408a10a\") "
Nov 03 10:45:13.790945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.783724    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-openshift-ca" pod "6f56a720-bdac-485d-ae61-f983f408a10a" (UID: "6f56a720-bdac-485d-ae61-f983f408a10a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:13.803701 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.801359    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-xks-ca" pod "6f56a720-bdac-485d-ae61-f983f408a10a" (UID: "6f56a720-bdac-485d-ae61-f983f408a10a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:13.807023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.804455    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f56a720-bdac-485d-ae61-f983f408a10a-kube-api-access-7trjd" pod "6f56a720-bdac-485d-ae61-f983f408a10a" (UID: "6f56a720-bdac-485d-ae61-f983f408a10a"). InnerVolumeSpecName "kube-api-access-7trjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:13.884310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.884265    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:13.884310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.884303    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trjd\" (UniqueName: \"kubernetes.io/projected/6f56a720-bdac-485d-ae61-f983f408a10a-kube-api-access-7trjd\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:13.884310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:13.884319    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f56a720-bdac-485d-ae61-f983f408a10a-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:14.572304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:14.572209    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r" event={"ID":"6f56a720-bdac-485d-ae61-f983f408a10a","Type":"ContainerDied","Data":"83208f439f9692a25e3d863059f0953c8520eefc8caba01feeb284cc755b9b87"}
Nov 03 10:45:14.572304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:14.572241    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"
Nov 03 10:45:14.572304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:14.572296    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83208f439f9692a25e3d863059f0953c8520eefc8caba01feeb284cc755b9b87"
Nov 03 10:45:14.751523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:14.751489    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 10:45:14.751688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:14.751541    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 10:45:15.749626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:15.749588    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 10:45:15.750087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:15.749661    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 10:45:16.142341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.142305    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"]
Nov 03 10:45:16.145694 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.145668    5393 cpu_manager.go:414] "containerMap: removing container" podUID="6f56a720-bdac-485d-ae61-f983f408a10a" containerName="revoke-keys"
Nov 03 10:45:16.145937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.145898    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f56a720-bdac-485d-ae61-f983f408a10a" containerName="revoke-keys"
Nov 03 10:45:16.146229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.146214    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f56a720-bdac-485d-ae61-f983f408a10a" containerName="revoke-keys"
Nov 03 10:45:16.168257 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.168231    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.173424 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.173397    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:16.198830 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.198803    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"]
Nov 03 10:45:16.353817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.353779    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.354010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.353834    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjmd\" (UniqueName: \"kubernetes.io/projected/ac8da127-97af-43f8-b6b2-3dd18a12b072-kube-api-access-qjjmd\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.354072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.354046    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.455104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.455015    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.455282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.455131    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.455282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.455169    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjmd\" (UniqueName: \"kubernetes.io/projected/ac8da127-97af-43f8-b6b2-3dd18a12b072-kube-api-access-qjjmd\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.455756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.455736    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.457693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.457676    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:16.463843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.463810    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjmd\" (UniqueName: \"kubernetes.io/projected/ac8da127-97af-43f8-b6b2-3dd18a12b072-kube-api-access-qjjmd\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.468399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.468378    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96b3cb37-n4b8z\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.549409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.549373    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:16.750209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.750131    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 10:45:16.750209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.750181    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 10:45:16.850671 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:16.850637    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"]
Nov 03 10:45:16.851952 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:45:16.851906    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8da127_97af_43f8_b6b2_3dd18a12b072.slice/crio-4225796702c2670ec2f996ca99a8b8f4cb549b21bf51b0a12170d5ed89e58634 WatchSource:0}: Error finding container 4225796702c2670ec2f996ca99a8b8f4cb549b21bf51b0a12170d5ed89e58634: Status 404 returned error can't find the container with id 4225796702c2670ec2f996ca99a8b8f4cb549b21bf51b0a12170d5ed89e58634
Nov 03 10:45:17.223292 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:17.223264    5393 scope.go:122] "RemoveContainer" containerID="e46a9f33fdb280bb3b1c8f701cf7d8abb9ead7aca5e0fef17ff53a63fa26f653"
Nov 03 10:45:17.624482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:17.624445    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z" event={"ID":"ac8da127-97af-43f8-b6b2-3dd18a12b072","Type":"ContainerStarted","Data":"d51ba8d68eda65c7da5097a72a514ce00aca75a01b1e05b3bea50906867acede"}
Nov 03 10:45:17.624624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:17.624517    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z" event={"ID":"ac8da127-97af-43f8-b6b2-3dd18a12b072","Type":"ContainerStarted","Data":"4225796702c2670ec2f996ca99a8b8f4cb549b21bf51b0a12170d5ed89e58634"}
Nov 03 10:45:17.644101 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:17.644046    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z" podStartSLOduration=1.644030772 podStartE2EDuration="1.644030772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:17.639426195 +0000 UTC m=+961.544280635" watchObservedRunningTime="2026-11-03 10:45:17.644030772 +0000 UTC m=+961.548885251"
Nov 03 10:45:17.749331 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:17.749295    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 10:45:17.749502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:17.749352    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 10:45:18.657638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:18.657517    5393 generic.go:356] "Generic (PLEG): container finished" podID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerID="7beb5a5e50e9d06dfb23c89c75816908ded6060edffbed42630bb5ef58a9377d" exitCode=60
Nov 03 10:45:18.658227 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:18.657642    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" event={"ID":"3d4cc4b8-ef31-416f-952a-7c4cfc54c411","Type":"ContainerDied","Data":"7beb5a5e50e9d06dfb23c89c75816908ded6060edffbed42630bb5ef58a9377d"}
Nov 03 10:45:18.658227 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:18.657704    5393 scope.go:122] "RemoveContainer" containerID="e46a9f33fdb280bb3b1c8f701cf7d8abb9ead7aca5e0fef17ff53a63fa26f653"
Nov 03 10:45:18.658359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:18.658253    5393 scope.go:122] "RemoveContainer" containerID="7beb5a5e50e9d06dfb23c89c75816908ded6060edffbed42630bb5ef58a9377d"
Nov 03 10:45:18.659403 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:18.658730    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842_odh-ai-gateway-infra(3d4cc4b8-ef31-416f-952a-7c4cfc54c411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411"
Nov 03 10:45:18.694199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:18.694162    5393 generic.go:356] "Generic (PLEG): container finished" podID="ac8da127-97af-43f8-b6b2-3dd18a12b072" containerID="d51ba8d68eda65c7da5097a72a514ce00aca75a01b1e05b3bea50906867acede" exitCode=0
Nov 03 10:45:18.694529 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:18.694506    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z" event={"ID":"ac8da127-97af-43f8-b6b2-3dd18a12b072","Type":"ContainerDied","Data":"d51ba8d68eda65c7da5097a72a514ce00aca75a01b1e05b3bea50906867acede"}
Nov 03 10:45:18.749719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:18.749686    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 10:45:18.749836 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:18.749746    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 10:45:19.223513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:19.223483    5393 scope.go:122] "RemoveContainer" containerID="108b65b83a353660d0d00a148e322241fc123d4e9d2745d6a4cb9be473a4316c"
Nov 03 10:45:19.688833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:19.688789    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"]
Nov 03 10:45:19.713963 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:19.713899    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" event={"ID":"ad3797e5-c99a-4c21-bfd3-45a70ee032bd","Type":"ContainerStarted","Data":"6b522d6b7822d55033635c32cfe68e783cd31fd9edb4600929808a08fbbd7993"}
Nov 03 10:45:19.791054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:19.791018    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 10:45:19.792510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:19.792474    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 10:45:20.655945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.654134    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:20.817950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.810492    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 10:45:20.817950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.810566    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 10:45:20.899274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.899226    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:20.909947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.909830    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842" event={"ID":"3d4cc4b8-ef31-416f-952a-7c4cfc54c411","Type":"ContainerDied","Data":"5c9971d4a7139b2b033c13722070f4f5fdc3c9b44b961a2312532ed540287a18"}
Nov 03 10:45:20.933152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.933122    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" event={"ID":"ad3797e5-c99a-4c21-bfd3-45a70ee032bd","Type":"ContainerDied","Data":"6b522d6b7822d55033635c32cfe68e783cd31fd9edb4600929808a08fbbd7993"}
Nov 03 10:45:20.935120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.923488    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-maas-api-service-ca\") pod \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\" (UID: \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\") "
Nov 03 10:45:20.935120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.933695    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-kube-api-access-4tmmn\" (UniqueName: \"kubernetes.io/projected/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-kube-api-access-4tmmn\") pod \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\" (UID: \"3d4cc4b8-ef31-416f-952a-7c4cfc54c411\") "
Nov 03 10:45:20.935120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.932904    5393 generic.go:356] "Generic (PLEG): container finished" podID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerID="6b522d6b7822d55033635c32cfe68e783cd31fd9edb4600929808a08fbbd7993" exitCode=60
Nov 03 10:45:20.935120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.910224    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"
Nov 03 10:45:20.935120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.933364    5393 scope.go:122] "RemoveContainer" containerID="7beb5a5e50e9d06dfb23c89c75816908ded6060edffbed42630bb5ef58a9377d"
Nov 03 10:45:20.935120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.924190    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-maas-api-service-ca" pod "3d4cc4b8-ef31-416f-952a-7c4cfc54c411" (UID: "3d4cc4b8-ef31-416f-952a-7c4cfc54c411"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:20.935748 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:20.935732    5393 scope.go:122] "RemoveContainer" containerID="6b522d6b7822d55033635c32cfe68e783cd31fd9edb4600929808a08fbbd7993"
Nov 03 10:45:20.936197 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:20.936171    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7_odh-ai-gateway-infra(ad3797e5-c99a-4c21-bfd3-45a70ee032bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd"
Nov 03 10:45:21.081944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.056691    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:21.145586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.142958    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"]
Nov 03 10:45:21.156326 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.156271    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-kube-api-access-4tmmn" pod "3d4cc4b8-ef31-416f-952a-7c4cfc54c411" (UID: "3d4cc4b8-ef31-416f-952a-7c4cfc54c411"). InnerVolumeSpecName "kube-api-access-4tmmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:21.234996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.212751    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmmn\" (UniqueName: \"kubernetes.io/projected/3d4cc4b8-ef31-416f-952a-7c4cfc54c411-kube-api-access-4tmmn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:21.237409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.235836    5393 scope.go:122] "RemoveContainer" containerID="3cc00b82d093955624e89bfe04dd5391e57a1341a8eb88fe90c86e5c3a46b0fc"
Nov 03 10:45:21.241208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.241115    5393 scope.go:122] "RemoveContainer" containerID="108b65b83a353660d0d00a148e322241fc123d4e9d2745d6a4cb9be473a4316c"
Nov 03 10:45:21.300995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.299356    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:45:21.301284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.301260    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:45:21.347828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.347040    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:21.348177 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.348148    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:21.478944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.475688    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:21.516595 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.516076    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:21.542741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.542708    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-openshift-ca\") pod \"ac8da127-97af-43f8-b6b2-3dd18a12b072\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") "
Nov 03 10:45:21.543130 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.543106    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-xks-ca\") pod \"ac8da127-97af-43f8-b6b2-3dd18a12b072\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") "
Nov 03 10:45:21.544282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.543371    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8da127-97af-43f8-b6b2-3dd18a12b072-kube-api-access-qjjmd\" (UniqueName: \"kubernetes.io/projected/ac8da127-97af-43f8-b6b2-3dd18a12b072-kube-api-access-qjjmd\") pod \"ac8da127-97af-43f8-b6b2-3dd18a12b072\" (UID: \"ac8da127-97af-43f8-b6b2-3dd18a12b072\") "
Nov 03 10:45:21.564646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.564605    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-openshift-ca" pod "ac8da127-97af-43f8-b6b2-3dd18a12b072" (UID: "ac8da127-97af-43f8-b6b2-3dd18a12b072"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:21.566122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.566099    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:21.778998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.772352    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-xks-ca" pod "ac8da127-97af-43f8-b6b2-3dd18a12b072" (UID: "ac8da127-97af-43f8-b6b2-3dd18a12b072"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:21.778998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.772450    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 10:45:21.778998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.772499    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 10:45:21.778998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.772994    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8da127-97af-43f8-b6b2-3dd18a12b072-kube-api-access-qjjmd" pod "ac8da127-97af-43f8-b6b2-3dd18a12b072" (UID: "ac8da127-97af-43f8-b6b2-3dd18a12b072"). InnerVolumeSpecName "kube-api-access-qjjmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:21.789441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.789364    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"]
Nov 03 10:45:21.820307 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.819841    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895045-mt842"]
Nov 03 10:45:21.835959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.835590    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjmd\" (UniqueName: \"kubernetes.io/projected/ac8da127-97af-43f8-b6b2-3dd18a12b072-kube-api-access-qjjmd\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:21.835959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.835629    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8da127-97af-43f8-b6b2-3dd18a12b072-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:21.906946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.905032    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:21.906946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:21.906285    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:45:22.036563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.036473    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z" event={"ID":"ac8da127-97af-43f8-b6b2-3dd18a12b072","Type":"ContainerDied","Data":"4225796702c2670ec2f996ca99a8b8f4cb549b21bf51b0a12170d5ed89e58634"}
Nov 03 10:45:22.036801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.036783    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4225796702c2670ec2f996ca99a8b8f4cb549b21bf51b0a12170d5ed89e58634"
Nov 03 10:45:22.055063 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.054911    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"
Nov 03 10:45:22.335118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.335053    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.383945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.381394    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" path="/var/lib/kubelet/pods/3d4cc4b8-ef31-416f-952a-7c4cfc54c411/volumes"
Nov 03 10:45:22.383945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.383173    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.402962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.392123    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:45:22.402962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.392182    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:45:22.794945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.786612    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.911960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.909608    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:22.912965 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.912835    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:22.912965 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:22.912885    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.047165 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.047068    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.076813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.076771    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.159969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.158290    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.159969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.158671    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:45:23.295962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.295232    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:45:23.296239 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.296211    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:23.296439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.296413    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.336882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.336543    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" event={"ID":"0454f4d8-c98f-4eb2-84b1-cb55441c6709","Type":"ContainerStarted","Data":"f0c8daa5f0c07c944c77202342e7d132c4afa97b5fe99071375f3206ded83ffa"}
Nov 03 10:45:23.337067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.337037    5393 scope.go:122] "RemoveContainer" containerID="f0c8daa5f0c07c944c77202342e7d132c4afa97b5fe99071375f3206ded83ffa"
Nov 03 10:45:23.337413 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:23.337385    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895045-57bxv_odh-ai-gateway-infra(0454f4d8-c98f-4eb2-84b1-cb55441c6709)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709"
Nov 03 10:45:23.338365 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.338341    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:45:23.403945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.385438    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.504179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.501991    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.506860 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.505953    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.701853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.701824    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:45:23.826507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.826468    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:45:23.826771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.826748    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:23.868945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.854274    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:45:23.870207 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.870173    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:45:23.950380 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:23.950349    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:45:24.081077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.078788    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.081256 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.081070    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.102720    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"]
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103518    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103538    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103576    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103584    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103597    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ac8da127-97af-43f8-b6b2-3dd18a12b072" containerName="revoke-keys"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103605    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8da127-97af-43f8-b6b2-3dd18a12b072" containerName="revoke-keys"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103718    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103733    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.103742    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8da127-97af-43f8-b6b2-3dd18a12b072" containerName="revoke-keys"
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.107758    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"]
Nov 03 10:45:24.110453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.107876    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.119942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.119239    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:24.170943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.170004    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.170943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.170679    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:45:24.321249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.321216    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:24.378957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.355802    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-kube-api-access-7jsq4\" (UniqueName: \"kubernetes.io/projected/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-kube-api-access-7jsq4\") pod \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\" (UID: \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\") "
Nov 03 10:45:24.522004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.436098    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-maas-api-service-ca\") pod \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\" (UID: \"ad3797e5-c99a-4c21-bfd3-45a70ee032bd\") "
Nov 03 10:45:24.522004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.436440    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.522004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.436754    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcd5l\" (UniqueName: \"kubernetes.io/projected/1c362709-1ebc-4d12-83ae-655af32fb700-kube-api-access-kcd5l\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.522004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.436817    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.522004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.433764    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-kube-api-access-7jsq4" pod "ad3797e5-c99a-4c21-bfd3-45a70ee032bd" (UID: "ad3797e5-c99a-4c21-bfd3-45a70ee032bd"). InnerVolumeSpecName "kube-api-access-7jsq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:24.522004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.457192    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-maas-api-service-ca" pod "ad3797e5-c99a-4c21-bfd3-45a70ee032bd" (UID: "ad3797e5-c99a-4c21-bfd3-45a70ee032bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:24.530613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.530526    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:24.532664 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.532632    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:45:24.532864 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.532852    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:45:24.570947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.570407    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcd5l\" (UniqueName: \"kubernetes.io/projected/1c362709-1ebc-4d12-83ae-655af32fb700-kube-api-access-kcd5l\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.570947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.570520    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.570947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.570677    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.570947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.570944    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:24.571393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.570966    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jsq4\" (UniqueName: \"kubernetes.io/projected/ad3797e5-c99a-4c21-bfd3-45a70ee032bd-kube-api-access-7jsq4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:24.606749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.597831    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.650950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.650646    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:45:24.650950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.650696    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"]
Nov 03 10:45:24.650950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.650758    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:45:24.702687 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.700796    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.702687 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.702204    5393 generic.go:356] "Generic (PLEG): container finished" podID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerID="f0c8daa5f0c07c944c77202342e7d132c4afa97b5fe99071375f3206ded83ffa" exitCode=60
Nov 03 10:45:24.702687 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.702355    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:45:24.702687 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.702395    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" event={"ID":"0454f4d8-c98f-4eb2-84b1-cb55441c6709","Type":"ContainerDied","Data":"f0c8daa5f0c07c944c77202342e7d132c4afa97b5fe99071375f3206ded83ffa"}
Nov 03 10:45:24.702687 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.702440    5393 scope.go:122] "RemoveContainer" containerID="3cc00b82d093955624e89bfe04dd5391e57a1341a8eb88fe90c86e5c3a46b0fc"
Nov 03 10:45:24.731799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.731669    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7" event={"ID":"ad3797e5-c99a-4c21-bfd3-45a70ee032bd","Type":"ContainerDied","Data":"017bb0455c96fc43109c975e43317879a9727b4e449a57569bcdfa007837c099"}
Nov 03 10:45:24.732101 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.732061    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"
Nov 03 10:45:24.775105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.775075    5393 scope.go:122] "RemoveContainer" containerID="6b522d6b7822d55033635c32cfe68e783cd31fd9edb4600929808a08fbbd7993"
Nov 03 10:45:24.806880 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.806303    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcd5l\" (UniqueName: \"kubernetes.io/projected/1c362709-1ebc-4d12-83ae-655af32fb700-kube-api-access-kcd5l\") pod \"maas-api-revoke-sub-8f575da8-hx6lh\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:24.806880 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.806699    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:24.808377 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.808285    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:24.835375 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.828043    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:45:24.894236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.894207    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"]
Nov 03 10:45:24.902195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.902159    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895045-nvsp7"]
Nov 03 10:45:24.902384 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:24.902368    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:25.050716 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.050686    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:25.134656 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.134621    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0454f4d8-c98f-4eb2-84b1-cb55441c6709-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0454f4d8-c98f-4eb2-84b1-cb55441c6709-maas-api-service-ca\") pod \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\" (UID: \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\") "
Nov 03 10:45:25.134798 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.134775    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0454f4d8-c98f-4eb2-84b1-cb55441c6709-kube-api-access-46c8r\" (UniqueName: \"kubernetes.io/projected/0454f4d8-c98f-4eb2-84b1-cb55441c6709-kube-api-access-46c8r\") pod \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\" (UID: \"0454f4d8-c98f-4eb2-84b1-cb55441c6709\") "
Nov 03 10:45:25.135630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.135575    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0454f4d8-c98f-4eb2-84b1-cb55441c6709-maas-api-service-ca" pod "0454f4d8-c98f-4eb2-84b1-cb55441c6709" (UID: "0454f4d8-c98f-4eb2-84b1-cb55441c6709"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:25.138024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.137995    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0454f4d8-c98f-4eb2-84b1-cb55441c6709-kube-api-access-46c8r" pod "0454f4d8-c98f-4eb2-84b1-cb55441c6709" (UID: "0454f4d8-c98f-4eb2-84b1-cb55441c6709"). InnerVolumeSpecName "kube-api-access-46c8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:25.236793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.236756    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0454f4d8-c98f-4eb2-84b1-cb55441c6709-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:25.236793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.236794    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46c8r\" (UniqueName: \"kubernetes.io/projected/0454f4d8-c98f-4eb2-84b1-cb55441c6709-kube-api-access-46c8r\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:25.309241 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:45:25.309209    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c362709_1ebc_4d12_83ae_655af32fb700.slice/crio-4d831ccb871da774b323c00ea7ad30183022e52e3403f2ea608c0cc013574c4d WatchSource:0}: Error finding container 4d831ccb871da774b323c00ea7ad30183022e52e3403f2ea608c0cc013574c4d: Status 404 returned error can't find the container with id 4d831ccb871da774b323c00ea7ad30183022e52e3403f2ea608c0cc013574c4d
Nov 03 10:45:25.309379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.309334    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"]
Nov 03 10:45:25.421399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.421366    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"]
Nov 03 10:45:25.422013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.421992    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:25.422013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422009    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422036    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422043    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422056    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422062    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422077    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422082    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422090    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422094    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422176    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422186    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422193    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422203    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4cc4b8-ef31-416f-952a-7c4cfc54c411" containerName="cleanup"
Nov 03 10:45:25.422217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.422214    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:25.427535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.427511    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.433423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.433388    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"]
Nov 03 10:45:25.540989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.540879    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkhtf\" (UniqueName: \"kubernetes.io/projected/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-kube-api-access-vkhtf\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.541154 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.541098    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.541301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.541271    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.642653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.642599    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkhtf\" (UniqueName: \"kubernetes.io/projected/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-kube-api-access-vkhtf\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.642867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.642731    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.642947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.642875    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.644744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.644702    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.646342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.646315    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.652294 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.652262    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkhtf\" (UniqueName: \"kubernetes.io/projected/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-kube-api-access-vkhtf\") pod \"maas-api-revoke-sub-33d476da-4csgp\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.748228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.748201    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"
Nov 03 10:45:25.748427 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.748235    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv" event={"ID":"0454f4d8-c98f-4eb2-84b1-cb55441c6709","Type":"ContainerDied","Data":"03ef4df82908a1c10c5210dfa83e024ba10f182e47e737b551822e4378c05e5f"}
Nov 03 10:45:25.748427 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.748309    5393 scope.go:122] "RemoveContainer" containerID="f0c8daa5f0c07c944c77202342e7d132c4afa97b5fe99071375f3206ded83ffa"
Nov 03 10:45:25.749798 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.749766    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh" event={"ID":"1c362709-1ebc-4d12-83ae-655af32fb700","Type":"ContainerStarted","Data":"4d831ccb871da774b323c00ea7ad30183022e52e3403f2ea608c0cc013574c4d"}
Nov 03 10:45:25.763721 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.763698    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:25.773882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.773845    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"]
Nov 03 10:45:25.777484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:25.777456    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895045-57bxv"]
Nov 03 10:45:26.135404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:26.132821    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"]
Nov 03 10:45:26.240954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:26.240898    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" path="/var/lib/kubelet/pods/0454f4d8-c98f-4eb2-84b1-cb55441c6709/volumes"
Nov 03 10:45:26.241844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:26.241819    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" path="/var/lib/kubelet/pods/ad3797e5-c99a-4c21-bfd3-45a70ee032bd/volumes"
Nov 03 10:45:26.761726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:26.761649    5393 generic.go:356] "Generic (PLEG): container finished" podID="1c362709-1ebc-4d12-83ae-655af32fb700" containerID="a827cc35e38fe4df4c3c92a33ceee5886deac22d2a28fea404b3250bac24fb31" exitCode=0
Nov 03 10:45:26.761898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:26.761740    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh" event={"ID":"1c362709-1ebc-4d12-83ae-655af32fb700","Type":"ContainerDied","Data":"a827cc35e38fe4df4c3c92a33ceee5886deac22d2a28fea404b3250bac24fb31"}
Nov 03 10:45:26.764150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:26.764112    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp" event={"ID":"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341","Type":"ContainerStarted","Data":"04ea4b89419d8ee7318a48841f81099e728e46773b02d53086fba49fbf7c8325"}
Nov 03 10:45:26.764273 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:26.764174    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp" event={"ID":"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341","Type":"ContainerStarted","Data":"081cac2f0e95f2b9fed6f4c5790c28fe4057df45d56753c73d3aaadb41b73a40"}
Nov 03 10:45:26.795869 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:26.795821    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp" podStartSLOduration=1.79580782 podStartE2EDuration="1.79580782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:26.793836289 +0000 UTC m=+970.698690732" watchObservedRunningTime="2026-11-03 10:45:26.79580782 +0000 UTC m=+970.700662260"
Nov 03 10:45:27.098179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.096089    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"]
Nov 03 10:45:27.098179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.096957    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:27.098179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.096976    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:27.098179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.097013    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:27.098179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.097022    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:27.098179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.097165    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3797e5-c99a-4c21-bfd3-45a70ee032bd" containerName="cleanup"
Nov 03 10:45:27.098179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.097178    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0454f4d8-c98f-4eb2-84b1-cb55441c6709" containerName="cleanup"
Nov 03 10:45:27.112744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.110384    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"]
Nov 03 10:45:27.112744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.110532    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.192231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.192205    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.192580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.192341    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.192580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.192460    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9phv4\" (UniqueName: \"kubernetes.io/projected/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-kube-api-access-9phv4\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.293445 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.293404    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9phv4\" (UniqueName: \"kubernetes.io/projected/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-kube-api-access-9phv4\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.293618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.293471    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.293618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.293567    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.294235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.294210    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.296180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.296158    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.301138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.301114    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9phv4\" (UniqueName: \"kubernetes.io/projected/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-kube-api-access-9phv4\") pod \"maas-api-revoke-sub-b9b26426-ctz7l\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.437808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.437724    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:27.721436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.721407    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"]
Nov 03 10:45:27.722321 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:45:27.722289    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4af4af_8ed8_448a_96ae_1ae14fe8f4b3.slice/crio-795ec5259797872ae3ce9ccb87b2bf0c39afcd33775eee10d953f856e7f31528 WatchSource:0}: Error finding container 795ec5259797872ae3ce9ccb87b2bf0c39afcd33775eee10d953f856e7f31528: Status 404 returned error can't find the container with id 795ec5259797872ae3ce9ccb87b2bf0c39afcd33775eee10d953f856e7f31528
Nov 03 10:45:27.805026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.797274    5393 generic.go:356] "Generic (PLEG): container finished" podID="72fc1a45-3ec7-49a7-ba99-6bcf03ae4341" containerID="04ea4b89419d8ee7318a48841f81099e728e46773b02d53086fba49fbf7c8325" exitCode=0
Nov 03 10:45:27.805026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.797557    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp" event={"ID":"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341","Type":"ContainerDied","Data":"04ea4b89419d8ee7318a48841f81099e728e46773b02d53086fba49fbf7c8325"}
Nov 03 10:45:27.805026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:27.801414    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l" event={"ID":"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3","Type":"ContainerStarted","Data":"795ec5259797872ae3ce9ccb87b2bf0c39afcd33775eee10d953f856e7f31528"}
Nov 03 10:45:28.020278 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.020250    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:28.109177 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.109144    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-xks-ca\") pod \"1c362709-1ebc-4d12-83ae-655af32fb700\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") "
Nov 03 10:45:28.109335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.109198    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-openshift-ca\") pod \"1c362709-1ebc-4d12-83ae-655af32fb700\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") "
Nov 03 10:45:28.109426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.109340    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c362709-1ebc-4d12-83ae-655af32fb700-kube-api-access-kcd5l\" (UniqueName: \"kubernetes.io/projected/1c362709-1ebc-4d12-83ae-655af32fb700-kube-api-access-kcd5l\") pod \"1c362709-1ebc-4d12-83ae-655af32fb700\" (UID: \"1c362709-1ebc-4d12-83ae-655af32fb700\") "
Nov 03 10:45:28.109590 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.109562    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-openshift-ca" pod "1c362709-1ebc-4d12-83ae-655af32fb700" (UID: "1c362709-1ebc-4d12-83ae-655af32fb700"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:28.109856 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.109840    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.111989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.111964    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-xks-ca" pod "1c362709-1ebc-4d12-83ae-655af32fb700" (UID: "1c362709-1ebc-4d12-83ae-655af32fb700"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:28.112070 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.112018    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c362709-1ebc-4d12-83ae-655af32fb700-kube-api-access-kcd5l" pod "1c362709-1ebc-4d12-83ae-655af32fb700" (UID: "1c362709-1ebc-4d12-83ae-655af32fb700"). InnerVolumeSpecName "kube-api-access-kcd5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:28.211461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.211435    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c362709-1ebc-4d12-83ae-655af32fb700-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.211461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.211461    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcd5l\" (UniqueName: \"kubernetes.io/projected/1c362709-1ebc-4d12-83ae-655af32fb700-kube-api-access-kcd5l\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:28.809847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.809824    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"
Nov 03 10:45:28.809997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.809823    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh" event={"ID":"1c362709-1ebc-4d12-83ae-655af32fb700","Type":"ContainerDied","Data":"4d831ccb871da774b323c00ea7ad30183022e52e3403f2ea608c0cc013574c4d"}
Nov 03 10:45:28.810062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.809996    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d831ccb871da774b323c00ea7ad30183022e52e3403f2ea608c0cc013574c4d"
Nov 03 10:45:28.811434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.811409    5393 generic.go:356] "Generic (PLEG): container finished" podID="1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3" containerID="48afe8e6b4159b1fbc2a94a4fb431c6797256dc97aa01dcad0bd62737c00553e" exitCode=0
Nov 03 10:45:28.811518 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.811487    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l" event={"ID":"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3","Type":"ContainerDied","Data":"48afe8e6b4159b1fbc2a94a4fb431c6797256dc97aa01dcad0bd62737c00553e"}
Nov 03 10:45:28.972278 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:28.972249    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:29.027203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.027163    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-openshift-ca\") pod \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") "
Nov 03 10:45:29.027432 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.027408    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-kube-api-access-vkhtf\" (UniqueName: \"kubernetes.io/projected/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-kube-api-access-vkhtf\") pod \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") "
Nov 03 10:45:29.027507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.027492    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-xks-ca\") pod \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\" (UID: \"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341\") "
Nov 03 10:45:29.027605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.027575    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-openshift-ca" pod "72fc1a45-3ec7-49a7-ba99-6bcf03ae4341" (UID: "72fc1a45-3ec7-49a7-ba99-6bcf03ae4341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:29.028154 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.028062    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:29.029876 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.029851    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-kube-api-access-vkhtf" pod "72fc1a45-3ec7-49a7-ba99-6bcf03ae4341" (UID: "72fc1a45-3ec7-49a7-ba99-6bcf03ae4341"). InnerVolumeSpecName "kube-api-access-vkhtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:29.030111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.030009    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-xks-ca" pod "72fc1a45-3ec7-49a7-ba99-6bcf03ae4341" (UID: "72fc1a45-3ec7-49a7-ba99-6bcf03ae4341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:29.129896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.129849    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkhtf\" (UniqueName: \"kubernetes.io/projected/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-kube-api-access-vkhtf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:29.129896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.129899    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:29.820456 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.820426    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"
Nov 03 10:45:29.820904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.820426    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp" event={"ID":"72fc1a45-3ec7-49a7-ba99-6bcf03ae4341","Type":"ContainerDied","Data":"081cac2f0e95f2b9fed6f4c5790c28fe4057df45d56753c73d3aaadb41b73a40"}
Nov 03 10:45:29.820904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.820585    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="081cac2f0e95f2b9fed6f4c5790c28fe4057df45d56753c73d3aaadb41b73a40"
Nov 03 10:45:29.828390 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.828363    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"]
Nov 03 10:45:29.829028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.829009    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1c362709-1ebc-4d12-83ae-655af32fb700" containerName="revoke-keys"
Nov 03 10:45:29.829028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.829027    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c362709-1ebc-4d12-83ae-655af32fb700" containerName="revoke-keys"
Nov 03 10:45:29.829210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.829041    5393 cpu_manager.go:414] "containerMap: removing container" podUID="72fc1a45-3ec7-49a7-ba99-6bcf03ae4341" containerName="revoke-keys"
Nov 03 10:45:29.829210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.829049    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fc1a45-3ec7-49a7-ba99-6bcf03ae4341" containerName="revoke-keys"
Nov 03 10:45:29.829210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.829183    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c362709-1ebc-4d12-83ae-655af32fb700" containerName="revoke-keys"
Nov 03 10:45:29.829210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.829194    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fc1a45-3ec7-49a7-ba99-6bcf03ae4341" containerName="revoke-keys"
Nov 03 10:45:29.833825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.833637    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.847120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.846750    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"]
Nov 03 10:45:29.944349 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944290    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.944349 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944346    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lpm\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-kube-api-access-58lpm\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.944592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944393    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.944592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944428    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b295b2f8-4f1e-48e8-af11-40899d42e907-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.944592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944457    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.944592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944561    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.944819 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944602    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.944819 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944666    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:29.944819 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:29.944705    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.010128 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.010100    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:30.046058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046021    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-openshift-ca\") pod \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") "
Nov 03 10:45:30.046240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046142    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-xks-ca\") pod \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") "
Nov 03 10:45:30.046240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046224    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-kube-api-access-9phv4\" (UniqueName: \"kubernetes.io/projected/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-kube-api-access-9phv4\") pod \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\" (UID: \"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3\") "
Nov 03 10:45:30.046462 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046436    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.046589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046518    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.046653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046592    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.046653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046634    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lpm\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-kube-api-access-58lpm\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.046771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046687    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.046771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046739    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b295b2f8-4f1e-48e8-af11-40899d42e907-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.046881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046771    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.046881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046794    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-openshift-ca" pod "1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3" (UID: "1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:30.047037 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.046846    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.047037 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.047012    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.047206 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.047183    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:30.047544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.047520    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.048020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.047997    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-credential-socket\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.048444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.048403    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-certs\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.048760 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.048739    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-data\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.049347 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.049311    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b295b2f8-4f1e-48e8-af11-40899d42e907-istiod-ca-cert\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.050281 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.050235    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-envoy\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.050535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.050498    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-kube-api-access-9phv4" pod "1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3" (UID: "1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3"). InnerVolumeSpecName "kube-api-access-9phv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:30.051209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.051181    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-podinfo\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.057596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.057569    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-token\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.057784 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.057762    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lpm\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-kube-api-access-58lpm\") pod \"e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") " pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.058381 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.058352    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-xks-ca" pod "1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3" (UID: "1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:30.148828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.148793    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:30.148828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.148831    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9phv4\" (UniqueName: \"kubernetes.io/projected/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3-kube-api-access-9phv4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:30.163795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.163759    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:30.545502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.545440    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"]
Nov 03 10:45:30.603449 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.603413    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:30.603596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.603483    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:30.603596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.603514    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:30.831136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.830165    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l" event={"ID":"1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3","Type":"ContainerDied","Data":"795ec5259797872ae3ce9ccb87b2bf0c39afcd33775eee10d953f856e7f31528"}
Nov 03 10:45:30.831136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.830230    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="795ec5259797872ae3ce9ccb87b2bf0c39afcd33775eee10d953f856e7f31528"
Nov 03 10:45:30.831136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.830306    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"
Nov 03 10:45:30.834108 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.834084    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" event={"ID":"b295b2f8-4f1e-48e8-af11-40899d42e907","Type":"ContainerStarted","Data":"073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b"}
Nov 03 10:45:30.834251 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.834240    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" event={"ID":"b295b2f8-4f1e-48e8-af11-40899d42e907","Type":"ContainerStarted","Data":"97c02643512d2ad65ad99039dc367de17402c15f8b0a5070557ac158f9e29c0c"}
Nov 03 10:45:30.859417 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:30.859366    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podStartSLOduration=1.859346082 podStartE2EDuration="1.859346082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:30.854438093 +0000 UTC m=+974.759292536" watchObservedRunningTime="2026-11-03 10:45:30.859346082 +0000 UTC m=+974.764200523"
Nov 03 10:45:31.166346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:31.166226    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:31.166710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:31.166681    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 10:45:31.166879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:31.166849    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 10:45:32.199950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:32.193055    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 10:45:32.199950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:32.193122    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 10:45:33.164396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.164357    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 10:45:33.164682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.164651    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 10:45:33.581996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.581758    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"]
Nov 03 10:45:33.584571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.582614    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3" containerName="revoke-keys"
Nov 03 10:45:33.584571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.582639    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3" containerName="revoke-keys"
Nov 03 10:45:33.584571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.582762    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3" containerName="revoke-keys"
Nov 03 10:45:33.653259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.637011    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:33.684955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.684197    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"]
Nov 03 10:45:33.684955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.684330    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:33.697545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.695392    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:45:33.697545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.695535    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:33.768947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.761773    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:33.845002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.819965    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85n5\" (UniqueName: \"kubernetes.io/projected/17d39b59-b897-49c4-8b72-6238db5244b7-kube-api-access-s85n5\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:33.895945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:33.870808    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:34.007773 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.007731    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:34.023946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.008135    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:34.024422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.024394    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85n5\" (UniqueName: \"kubernetes.io/projected/17d39b59-b897-49c4-8b72-6238db5244b7-kube-api-access-s85n5\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:34.028658 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.028627    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:34.028881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.009902    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:34.061976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.057637    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85n5\" (UniqueName: \"kubernetes.io/projected/17d39b59-b897-49c4-8b72-6238db5244b7-kube-api-access-s85n5\") pod \"maas-api-revoke-sub-607e9ee7-vbsbf\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:34.164440 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.164246    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 10:45:34.165290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.165256    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 10:45:34.250077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.250040    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:34.555474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:34.555448    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"]
Nov 03 10:45:34.556795 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:45:34.556770    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d39b59_b897_49c4_8b72_6238db5244b7.slice/crio-6fc80328ed37156395524260ce943c007c1c0d94d0539a01d43c106322a9e0ab WatchSource:0}: Error finding container 6fc80328ed37156395524260ce943c007c1c0d94d0539a01d43c106322a9e0ab: Status 404 returned error can't find the container with id 6fc80328ed37156395524260ce943c007c1c0d94d0539a01d43c106322a9e0ab
Nov 03 10:45:35.075331 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:35.075248    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf" event={"ID":"17d39b59-b897-49c4-8b72-6238db5244b7","Type":"ContainerStarted","Data":"6fc80328ed37156395524260ce943c007c1c0d94d0539a01d43c106322a9e0ab"}
Nov 03 10:45:35.164615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:35.164582    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 10:45:35.164782 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:35.164642    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 10:45:36.107088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.107037    5393 generic.go:356] "Generic (PLEG): container finished" podID="17d39b59-b897-49c4-8b72-6238db5244b7" containerID="ffbc4150186c3f50bb061fb51f469214ed1cb9aab3974727b79c4a49acb88143" exitCode=0
Nov 03 10:45:36.107526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.107268    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf" event={"ID":"17d39b59-b897-49c4-8b72-6238db5244b7","Type":"ContainerDied","Data":"ffbc4150186c3f50bb061fb51f469214ed1cb9aab3974727b79c4a49acb88143"}
Nov 03 10:45:36.164403 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.164370    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 10:45:36.164552 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.164420    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 10:45:36.300942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.295706    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"]
Nov 03 10:45:36.327712 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.327680    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.332498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.332467    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"]
Nov 03 10:45:36.340425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.340400    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-141053-openshift-default-dockercfg-2d6xl\""
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471273    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv88b\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-kube-api-access-vv88b\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471346    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-certs\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471382    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-token\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471445    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istiod-ca-cert\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471492    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-credential-socket\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471556    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-data\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471578    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-podinfo\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471636    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-socket\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.478012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.471675    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-envoy\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.572910    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv88b\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-kube-api-access-vv88b\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.573026    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-certs\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.573065    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-token\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.573126    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istiod-ca-cert\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.573178    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-credential-socket\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.573249    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-data\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.573271    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-podinfo\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.573330    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-socket\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.573373    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-envoy\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.575995    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-certs\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.576212    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-data\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.576724    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istiod-ca-cert\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.577940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.576965    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-credential-socket\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.618951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.590025    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-socket\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.618951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.597355    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-envoy\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.636264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.624888    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-token\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.636264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.626607    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-podinfo\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.636264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.633560    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv88b\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-kube-api-access-vv88b\") pod \"e2e-api-a-141053-openshift-default-6694f679f7-2vzmm\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") " pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:36.651638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:36.651605    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:37.182944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.172187    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"]
Nov 03 10:45:37.182944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.172313    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 10:45:37.182944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.172361    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 10:45:37.234073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.233431    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:37.234073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.233505    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:37.234073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.233534    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:45:37.430998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.420749    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:37.450614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.446045    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-openshift-ca\") pod \"17d39b59-b897-49c4-8b72-6238db5244b7\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") "
Nov 03 10:45:37.450614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.446147    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d39b59-b897-49c4-8b72-6238db5244b7-kube-api-access-s85n5\" (UniqueName: \"kubernetes.io/projected/17d39b59-b897-49c4-8b72-6238db5244b7-kube-api-access-s85n5\") pod \"17d39b59-b897-49c4-8b72-6238db5244b7\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") "
Nov 03 10:45:37.450614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.446201    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-xks-ca\") pod \"17d39b59-b897-49c4-8b72-6238db5244b7\" (UID: \"17d39b59-b897-49c4-8b72-6238db5244b7\") "
Nov 03 10:45:37.450614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.449107    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-openshift-ca" pod "17d39b59-b897-49c4-8b72-6238db5244b7" (UID: "17d39b59-b897-49c4-8b72-6238db5244b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:37.465221 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.460041    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d39b59-b897-49c4-8b72-6238db5244b7-kube-api-access-s85n5" pod "17d39b59-b897-49c4-8b72-6238db5244b7" (UID: "17d39b59-b897-49c4-8b72-6238db5244b7"). InnerVolumeSpecName "kube-api-access-s85n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:37.468772 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.466201    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-xks-ca" pod "17d39b59-b897-49c4-8b72-6238db5244b7" (UID: "17d39b59-b897-49c4-8b72-6238db5244b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:37.556853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.553956    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:37.556853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.553999    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85n5\" (UniqueName: \"kubernetes.io/projected/17d39b59-b897-49c4-8b72-6238db5244b7-kube-api-access-s85n5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:37.556853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:37.554016    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17d39b59-b897-49c4-8b72-6238db5244b7-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:38.147505 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.147471    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"
Nov 03 10:45:38.148067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.148032    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf" event={"ID":"17d39b59-b897-49c4-8b72-6238db5244b7","Type":"ContainerDied","Data":"6fc80328ed37156395524260ce943c007c1c0d94d0539a01d43c106322a9e0ab"}
Nov 03 10:45:38.148211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.148144    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fc80328ed37156395524260ce943c007c1c0d94d0539a01d43c106322a9e0ab"
Nov 03 10:45:38.152970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.152944    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" event={"ID":"4103a55e-db0d-4d1f-9c9b-1e49d1102c65","Type":"ContainerStarted","Data":"c961e6abb66d2f3df226c323441930fbd22fc8a8513d349e27d24faae8f2f741"}
Nov 03 10:45:38.153071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.152987    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" event={"ID":"4103a55e-db0d-4d1f-9c9b-1e49d1102c65","Type":"ContainerStarted","Data":"eaf9a3af4dbc4f679478a0ffd3cc36bdf23f675627ab776cf9729343238d4f0f"}
Nov 03 10:45:38.164419 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.164393    5393 patch_prober.go:29] interesting pod/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 10:45:38.164561 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.164464    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 10:45:38.184351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.184274    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" podStartSLOduration=2.184254384 podStartE2EDuration="2.184254384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:38.175014895 +0000 UTC m=+982.079869338" watchObservedRunningTime="2026-11-03 10:45:38.184254384 +0000 UTC m=+982.089108822"
Nov 03 10:45:38.653363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.653334    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:38.659300 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:38.659273    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:39.161639 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:39.161612    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:39.162985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:39.162963    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:45:39.228245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:39.228216    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:39.228884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:39.228865    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:39.229995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:39.229972    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:45:43.152941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.145038    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.443566 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.442152    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:45:43.459086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.459037    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.498513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.498469    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:45:43.499523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.499487    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.568659 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:45:43.568617    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:45:43.691122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.691062    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.740196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.740067    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.764653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.764041    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:43.916075 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:43.916009    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.100954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.097014    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.161944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.149038    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.161944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.150062    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:45:44.469309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.462071    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.470379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.470350    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:45:44.505944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.484941    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.520652 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.520622    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:45:44.521755 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.505961    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.522253 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.505993    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 10:45:44.532473 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.522427    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:45:44.532636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.532512    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:45:44.758707 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.758339    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.759255 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.759220    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.760824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.759993    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:45:44.761087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.761072    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:45:44.784950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.781220    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.810083 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.809985    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:45:44.918604 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.917584    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:44.918604 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.918020    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:45:44.933101 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.933066    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"]
Nov 03 10:45:44.936401 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.936374    5393 cpu_manager.go:414] "containerMap: removing container" podUID="17d39b59-b897-49c4-8b72-6238db5244b7" containerName="revoke-keys"
Nov 03 10:45:44.936592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.936575    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d39b59-b897-49c4-8b72-6238db5244b7" containerName="revoke-keys"
Nov 03 10:45:44.936907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:44.936890    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d39b59-b897-49c4-8b72-6238db5244b7" containerName="revoke-keys"
Nov 03 10:45:45.034631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.034526    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.079952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.079900    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:45:45.107976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.081131    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:45:45.183951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.182007    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.200695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.185000    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"]
Nov 03 10:45:45.200695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.185695    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.214531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.212755    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwsf\" (UniqueName: \"kubernetes.io/projected/a8fb66bc-0987-4331-acfe-861e694099a8-kube-api-access-bpwsf\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.214531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.213509    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.371047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.342829    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.371047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.348081    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.371047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.354971    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwsf\" (UniqueName: \"kubernetes.io/projected/a8fb66bc-0987-4331-acfe-861e694099a8-kube-api-access-bpwsf\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.429677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.429635    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.450023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.430078    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:45.526193 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.526139    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.530955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.527995    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:45.530955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.528451    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:45.550063 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.550014    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.657715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.657636    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwsf\" (UniqueName: \"kubernetes.io/projected/a8fb66bc-0987-4331-acfe-861e694099a8-kube-api-access-bpwsf\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.753596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.753554    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51f25692-x9dzz\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:45.765429 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.765389    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.766295 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.766258    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:45.825217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.815726    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 10:45:45.873212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:45.873171    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:46.184323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.184293    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:45:46.233943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.226241    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.233943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.228062    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.233943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.228425    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.527248 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.527193    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.576012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.574293    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.576012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.574662    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:45:46.769485 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.769329    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Nov 03 10:45:46.781090 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.779424    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:46.893114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.893081    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:45:46.900067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:46.900039    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:45:47.030792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:47.030717    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:45:47.073329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:47.073292    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:45:47.153236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:47.153208    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:45:47.163954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:47.157633    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:45:47.202955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:47.164615    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:45:47.215372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:47.214195    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:45:47.256899 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:47.256874    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:45:47.303187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:47.297170    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"]
Nov 03 10:45:48.268533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:48.268491    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz" event={"ID":"a8fb66bc-0987-4331-acfe-861e694099a8","Type":"ContainerStarted","Data":"b38735d7972402e9cb38b0f7072e767ac25ccad9e3cb880c70ac94109164857d"}
Nov 03 10:45:48.269061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:48.268602    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz" event={"ID":"a8fb66bc-0987-4331-acfe-861e694099a8","Type":"ContainerStarted","Data":"a1e9827aa2fd2a9d473d28f748931b50f27eba7c8f8982224aaf38763c662a31"}
Nov 03 10:45:48.288791 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:48.288744    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz" podStartSLOduration=4.288729475 podStartE2EDuration="4.288729475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:48.283911892 +0000 UTC m=+992.188766333" watchObservedRunningTime="2026-11-03 10:45:48.288729475 +0000 UTC m=+992.193583935"
Nov 03 10:45:49.276551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:49.276518    5393 generic.go:356] "Generic (PLEG): container finished" podID="a8fb66bc-0987-4331-acfe-861e694099a8" containerID="b38735d7972402e9cb38b0f7072e767ac25ccad9e3cb880c70ac94109164857d" exitCode=0
Nov 03 10:45:49.276977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:49.276585    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz" event={"ID":"a8fb66bc-0987-4331-acfe-861e694099a8","Type":"ContainerDied","Data":"b38735d7972402e9cb38b0f7072e767ac25ccad9e3cb880c70ac94109164857d"}
Nov 03 10:45:50.440323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.440292    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:50.668047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.668007    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-xks-ca\") pod \"a8fb66bc-0987-4331-acfe-861e694099a8\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") "
Nov 03 10:45:50.668202 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.668144    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-openshift-ca\") pod \"a8fb66bc-0987-4331-acfe-861e694099a8\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") "
Nov 03 10:45:50.668405 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.668385    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fb66bc-0987-4331-acfe-861e694099a8-kube-api-access-bpwsf\" (UniqueName: \"kubernetes.io/projected/a8fb66bc-0987-4331-acfe-861e694099a8-kube-api-access-bpwsf\") pod \"a8fb66bc-0987-4331-acfe-861e694099a8\" (UID: \"a8fb66bc-0987-4331-acfe-861e694099a8\") "
Nov 03 10:45:50.668528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.668499    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-openshift-ca" pod "a8fb66bc-0987-4331-acfe-861e694099a8" (UID: "a8fb66bc-0987-4331-acfe-861e694099a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:45:50.668868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.668849    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:50.670387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.670358    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-xks-ca" pod "a8fb66bc-0987-4331-acfe-861e694099a8" (UID: "a8fb66bc-0987-4331-acfe-861e694099a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:45:50.670534 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.670510    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fb66bc-0987-4331-acfe-861e694099a8-kube-api-access-bpwsf" pod "a8fb66bc-0987-4331-acfe-861e694099a8" (UID: "a8fb66bc-0987-4331-acfe-861e694099a8"). InnerVolumeSpecName "kube-api-access-bpwsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:45:50.769660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.769626    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fb66bc-0987-4331-acfe-861e694099a8-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:50.769660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:50.769658    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpwsf\" (UniqueName: \"kubernetes.io/projected/a8fb66bc-0987-4331-acfe-861e694099a8-kube-api-access-bpwsf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:45:51.294826 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:51.294797    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"
Nov 03 10:45:51.294826 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:51.294811    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz" event={"ID":"a8fb66bc-0987-4331-acfe-861e694099a8","Type":"ContainerDied","Data":"a1e9827aa2fd2a9d473d28f748931b50f27eba7c8f8982224aaf38763c662a31"}
Nov 03 10:45:51.295150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:51.294871    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e9827aa2fd2a9d473d28f748931b50f27eba7c8f8982224aaf38763c662a31"
Nov 03 10:45:55.749806 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.749768    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"]
Nov 03 10:45:55.750357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.750342    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a8fb66bc-0987-4331-acfe-861e694099a8" containerName="revoke-keys"
Nov 03 10:45:55.750404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.750359    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fb66bc-0987-4331-acfe-861e694099a8" containerName="revoke-keys"
Nov 03 10:45:55.750466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.750457    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fb66bc-0987-4331-acfe-861e694099a8" containerName="revoke-keys"
Nov 03 10:45:55.754679 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.754659    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:55.772820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.772789    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ab592834\"/\"kube-root-ca.crt\""
Nov 03 10:45:55.772820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.772793    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ab592834\"/\"default-dockercfg-4578v\""
Nov 03 10:45:55.773072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.772885    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ab592834\"/\"openshift-service-ca.crt\""
Nov 03 10:45:55.773072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.772900    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ab592834\"/\"e2e-lifecycle-model-ab592834-kserve-self-signed-certs\""
Nov 03 10:45:55.779051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.779024    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"]
Nov 03 10:45:55.938376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.938330    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-model-cache\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:55.938526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.938470    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f231c2ce-f4b1-48c3-829e-a10e26e22960-tls-certs\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:55.938526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.938504    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5vj\" (UniqueName: \"kubernetes.io/projected/f231c2ce-f4b1-48c3-829e-a10e26e22960-kube-api-access-tf5vj\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:55.938654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.938536    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-dshm\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:55.938654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.938622    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-home\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:55.938772 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:55.938749    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-tmp-dir\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.040997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.040484    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-model-cache\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.040997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.040589    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f231c2ce-f4b1-48c3-829e-a10e26e22960-tls-certs\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.040997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.040624    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5vj\" (UniqueName: \"kubernetes.io/projected/f231c2ce-f4b1-48c3-829e-a10e26e22960-kube-api-access-tf5vj\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.040997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.040658    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-dshm\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.040997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.040705    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-home\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.040997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.040769    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-tmp-dir\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.041448 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.041294    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-tmp-dir\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.049943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.041561    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-model-cache\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.049943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.045793    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f231c2ce-f4b1-48c3-829e-a10e26e22960-tls-certs\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.053950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.053732    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5vj\" (UniqueName: \"kubernetes.io/projected/f231c2ce-f4b1-48c3-829e-a10e26e22960-kube-api-access-tf5vj\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.054143 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.054115    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-home\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.064290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.064243    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-dshm\") pod \"e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") " pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.081063 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.081024    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:45:56.470226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:56.470199    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"]
Nov 03 10:45:56.470419 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:45:56.470393    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf231c2ce_f4b1_48c3_829e_a10e26e22960.slice/crio-a58f8bdbb19d09132af0cf52396a667d832d054fcece708f8aa88d505e5a4063 WatchSource:0}: Error finding container a58f8bdbb19d09132af0cf52396a667d832d054fcece708f8aa88d505e5a4063: Status 404 returned error can't find the container with id a58f8bdbb19d09132af0cf52396a667d832d054fcece708f8aa88d505e5a4063
Nov 03 10:45:57.363650 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:57.363617    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x" event={"ID":"f231c2ce-f4b1-48c3-829e-a10e26e22960","Type":"ContainerStarted","Data":"f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4"}
Nov 03 10:45:57.364058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:57.363658    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x" event={"ID":"f231c2ce-f4b1-48c3-829e-a10e26e22960","Type":"ContainerStarted","Data":"a58f8bdbb19d09132af0cf52396a667d832d054fcece708f8aa88d505e5a4063"}
Nov 03 10:45:57.380426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:57.380382    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x" podStartSLOduration=2.380368588 podStartE2EDuration="2.380368588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:45:57.379561332 +0000 UTC m=+1001.284415772" watchObservedRunningTime="2026-11-03 10:45:57.380368588 +0000 UTC m=+1001.285223029"
Nov 03 10:45:58.082211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:45:58.082174    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:46:02.188944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:02.188794    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.298242 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:02.289171    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.530304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:02.529024    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.530578 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:02.530546    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.562066 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:02.562012    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.779194 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:02.779141    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:02.816381 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:02.816261    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.073438 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.072258    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.196976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.177053    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.198191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.198062    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.303025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.218364    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.303025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.227033    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:46:03.307950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.307083    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.307950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.307474    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:46:03.607894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.604666    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.607894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.605155    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:46:03.607894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.606050    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.607894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.606611    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:46:03.677173 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.677121    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 10:46:03.678904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.678875    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:46:03.825944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.822907    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:03.825944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:03.825080    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:46:04.097316 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.096979    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded"
Nov 03 10:46:04.191123 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.188356    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.191123 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.188707    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:46:04.243110 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.243069    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.267810 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.267767    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:04.319764 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.316116    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:04.621040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.620982    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:04.643314 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.643275    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 10:46:04.657851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.657817    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:46:04.677869 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.677829    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:04.924655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.924552    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:04.941346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:04.927325    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.191003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.188897    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.251946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.248172    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:46:05.277471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.276645    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.318119 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.317832    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 10:46:05.555000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.549844    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 10:46:05.644944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.642477    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:05.644944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.643807    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 10:46:05.694941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.691387    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 10:46:05.796499 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.796451    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:46:05.810007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.803304    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:46:05.810007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.804480    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:46:05.839959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.833689    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:46:05.872238 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.871788    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:46:05.925323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.924399    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:46:05.929620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.929395    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:46:05.945033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:05.942794    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:46:06.083781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:06.083695    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:46:06.119791 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:06.119756    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:46:07.264939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:07.257779    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:46:11.966357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:11.966149    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"]
Nov 03 10:46:12.017036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.017004    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"]
Nov 03 10:46:12.017228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.017155    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.023987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.023904    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a917b1-openshift-default-dockercfg-8qrlw\""
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.096864    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istiod-ca-cert\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.096976    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlc64\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-kube-api-access-jlc64\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.097053    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-certs\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.097080    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-podinfo\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.097108    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-data\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.097219    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-envoy\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.097266    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-token\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.097303    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-credential-socket\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.117945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.097328    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-socket\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.200660    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istiod-ca-cert\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.200746    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlc64\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-kube-api-access-jlc64\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.200811    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-certs\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.200832    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-podinfo\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.200858    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-data\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.200986    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-envoy\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.201031    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-token\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.201087    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-credential-socket\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.201113    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-socket\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.201604    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-socket\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.202218    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istiod-ca-cert\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.202753    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-certs\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.208944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.207792    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-credential-socket\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.215942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.213724    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-data\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.221616 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.220181    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlc64\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-kube-api-access-jlc64\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.225250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.225220    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-envoy\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.295633 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.295564    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-podinfo\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.308804 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.308771    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-token\") pod \"e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") " pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.339523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.339246    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:12.656103 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:46:12.656064    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a48dd6f_07bc_4633_aae0_54d49adc6bea.slice/crio-4312b5b57628e70a2998e463bb116b48fd2974ae75997b64916772b17a4f1aee WatchSource:0}: Error finding container 4312b5b57628e70a2998e463bb116b48fd2974ae75997b64916772b17a4f1aee: Status 404 returned error can't find the container with id 4312b5b57628e70a2998e463bb116b48fd2974ae75997b64916772b17a4f1aee
Nov 03 10:46:12.657425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.657400    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"]
Nov 03 10:46:12.659027 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.658983    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:12.659143 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.659069    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:12.659143 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:12.659103    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:13.194825 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:46:13.194787    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:46:13.303252 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:13.303215    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" event={"ID":"9a48dd6f-07bc-4633-aae0-54d49adc6bea","Type":"ContainerStarted","Data":"10e36722a4b51331f119975123abe9cfaf46b32a01b4d32f06f017874f48500a"}
Nov 03 10:46:13.303450 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:13.303273    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" event={"ID":"9a48dd6f-07bc-4633-aae0-54d49adc6bea","Type":"ContainerStarted","Data":"4312b5b57628e70a2998e463bb116b48fd2974ae75997b64916772b17a4f1aee"}
Nov 03 10:46:13.341604 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:13.341558    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:13.343611 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:13.343558    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" podStartSLOduration=2.343544821 podStartE2EDuration="2.343544821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:13.339309045 +0000 UTC m=+1017.244163499" watchObservedRunningTime="2026-11-03 10:46:13.343544821 +0000 UTC m=+1017.248399262"
Nov 03 10:46:13.347852 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:13.347824    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:14.350987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:14.338725    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:14.350987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:14.340869    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:46:15.187542 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.187509    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"]
Nov 03 10:46:15.192508 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.192483    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.200703 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.200672    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"]
Nov 03 10:46:15.201265 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.201238    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ab592834-openshift-default-dockercfg-vq7pv\""
Nov 03 10:46:15.267318 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267286    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vcw9\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-kube-api-access-8vcw9\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.267493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267355    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.267493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267470    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.267622 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267510    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.267701 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267676    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.267781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267730    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.267868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267854    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.267910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267877    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fed644-0876-44be-ad2c-4ff36b228ca6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.267996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.267930    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.346057 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.346027    5393 generic.go:356] "Generic (PLEG): container finished" podID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerID="c961e6abb66d2f3df226c323441930fbd22fc8a8513d349e27d24faae8f2f741" exitCode=0
Nov 03 10:46:15.346221 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.346100    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" event={"ID":"4103a55e-db0d-4d1f-9c9b-1e49d1102c65","Type":"ContainerDied","Data":"c961e6abb66d2f3df226c323441930fbd22fc8a8513d349e27d24faae8f2f741"}
Nov 03 10:46:15.346731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.346710    5393 scope.go:122] "RemoveContainer" containerID="c961e6abb66d2f3df226c323441930fbd22fc8a8513d349e27d24faae8f2f741"
Nov 03 10:46:15.348222 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.348192    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.348329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.348259    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.348329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.348290    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.368888 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.368863    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fed644-0876-44be-ad2c-4ff36b228ca6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.368933    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.368996    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vcw9\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-kube-api-access-8vcw9\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369092    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369134    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369164    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369279    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369332    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369530    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369621    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fed644-0876-44be-ad2c-4ff36b228ca6-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369652    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369906 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369790    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.369906 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369856    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.370040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.369910    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.372175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.372151    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.372879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.372855    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.380077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.380019    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vcw9\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-kube-api-access-8vcw9\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.381632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.381507    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-rh242\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.527580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.527549    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:15.850993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.845789    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"]
Nov 03 10:46:15.926050 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:46:15.925558    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fed644_0876_44be_ad2c_4ff36b228ca6.slice/crio-c97ebf6b5729c7cdeee5c5165c951851f1659c85fbfbdfcc087a272b9c8592eb WatchSource:0}: Error finding container c97ebf6b5729c7cdeee5c5165c951851f1659c85fbfbdfcc087a272b9c8592eb: Status 404 returned error can't find the container with id c97ebf6b5729c7cdeee5c5165c951851f1659c85fbfbdfcc087a272b9c8592eb
Nov 03 10:46:15.932989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.932731    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.932989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.932817    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:15.932989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:15.932854    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:16.357471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:16.357435    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" event={"ID":"a8fed644-0876-44be-ad2c-4ff36b228ca6","Type":"ContainerStarted","Data":"6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb"}
Nov 03 10:46:16.357630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:16.357506    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" event={"ID":"a8fed644-0876-44be-ad2c-4ff36b228ca6","Type":"ContainerStarted","Data":"c97ebf6b5729c7cdeee5c5165c951851f1659c85fbfbdfcc087a272b9c8592eb"}
Nov 03 10:46:16.361498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:16.360965    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" event={"ID":"4103a55e-db0d-4d1f-9c9b-1e49d1102c65","Type":"ContainerStarted","Data":"5c339e369dce91b4570798d5558560efe06aaf9d8bc1b3700c17b8cef5560bef"}
Nov 03 10:46:16.416351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:16.416292    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podStartSLOduration=1.416271734 podStartE2EDuration="1.416271734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:16.378002862 +0000 UTC m=+1020.282857302" watchObservedRunningTime="2026-11-03 10:46:16.416271734 +0000 UTC m=+1020.321126176"
Nov 03 10:46:16.528434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:16.528406    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:16.653718 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:16.653637    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:46:16.653718 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:16.653672    5393 patch_prober.go:29] interesting pod/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 10:46:16.653926 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:16.653716    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 10:46:17.550945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:17.550039    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-rh242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 10:46:17.550945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:17.550115    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 10:46:17.656581 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:17.652809    5393 patch_prober.go:29] interesting pod/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 10:46:17.656581 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:17.652866    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 10:46:18.530946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:18.530311    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-rh242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 10:46:18.530946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:18.530370    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 10:46:18.653036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:18.653001    5393 patch_prober.go:29] interesting pod/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 10:46:18.653420 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:18.653053    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 10:46:19.529343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:19.529303    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-rh242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 10:46:19.529618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:19.529590    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 10:46:19.653067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:19.653028    5393 patch_prober.go:29] interesting pod/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 10:46:19.653481 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:19.653095    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 10:46:20.529979 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:20.529536    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-rh242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 10:46:20.529979 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:20.529594    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 10:46:20.662348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:20.662312    5393 patch_prober.go:29] interesting pod/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 10:46:20.662947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:20.662890    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 10:46:21.528951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:21.528330    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-rh242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 10:46:21.528951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:21.528394    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 10:46:21.653182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:21.653146    5393 patch_prober.go:29] interesting pod/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 10:46:21.653357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:21.653220    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 10:46:22.528912 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:22.528878    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-rh242 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 10:46:22.529335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:22.528961    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 10:46:22.660374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:22.660338    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:46:22.663070 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:22.663042    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:46:22.663245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:22.663178    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:46:23.533114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:23.533077    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:23.533502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:23.533366    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:23.534027 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:23.534006    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:46:25.939792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:25.939736    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:32.886829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:32.886742    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"]
Nov 03 10:46:32.897492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:32.897465    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:32.902498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:32.902358    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"]
Nov 03 10:46:32.908420 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:32.908396    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7d8392-openshift-default-dockercfg-gcgtt\""
Nov 03 10:46:33.013038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.012997    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-socket\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.013230 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.013056    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxtf\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-kube-api-access-7mxtf\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.013284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.013249    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-envoy\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.013331 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.013289    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-data\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.013404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.013386    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-credential-socket\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.013451 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.013430    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-podinfo\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.013620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.013605    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istiod-ca-cert\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.013674 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.013658    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-token\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.013773 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.013741    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-certs\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.115590 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.115554    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-socket\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.115795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.115597    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxtf\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-kube-api-access-7mxtf\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.115795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.115656    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-envoy\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.115795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.115675    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-data\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.115795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.115703    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-credential-socket\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.115795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.115738    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-podinfo\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.116104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.115878    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istiod-ca-cert\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.116104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.115987    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-token\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.116104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.116032    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-certs\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.116104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.116065    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-socket\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.116316 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.116172    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-credential-socket\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.116509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.116439    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-data\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.116701 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.116681    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istiod-ca-cert\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.116798 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.116777    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-certs\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.118154 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.118137    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-envoy\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.118509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.118490    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-podinfo\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.147590 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.146372    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxtf\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-kube-api-access-7mxtf\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.166535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.166499    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-token\") pod \"e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") " pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.220118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.220028    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:33.732269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.732231    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"]
Nov 03 10:46:33.733351 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:46:33.733324    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c991cb_9d1c_4377_b3e1_8e34e4eb25a6.slice/crio-5b15191b3f2fc0e41a9ed30f9eda70b65b8ca354877fbd76248632b1a6ce90fd WatchSource:0}: Error finding container 5b15191b3f2fc0e41a9ed30f9eda70b65b8ca354877fbd76248632b1a6ce90fd: Status 404 returned error can't find the container with id 5b15191b3f2fc0e41a9ed30f9eda70b65b8ca354877fbd76248632b1a6ce90fd
Nov 03 10:46:33.736017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.735982    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:33.736175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.736051    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:33.736175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:33.736081    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:34.171350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.168878    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"]
Nov 03 10:46:34.192073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.191377    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"]
Nov 03 10:46:34.192073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.191502    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.204600 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.203190    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-273012-openshift-default-dockercfg-8z8dj\""
Nov 03 10:46:34.267095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267000    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-envoy\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.267269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267146    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-certs\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.267269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267224    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-data\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.267269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267250    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-podinfo\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.267444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267328    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac04ed6c-cedf-460c-b512-a9a2380a9793-istiod-ca-cert\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.267444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267438    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-socket\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.267551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267498    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-token\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.267646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267629    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-credential-socket\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.267710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.267651    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q2dj\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-kube-api-access-6q2dj\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.369284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.369243    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-credential-socket\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.369514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.369493    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q2dj\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-kube-api-access-6q2dj\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.369683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.369666    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-envoy\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.369896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.369877    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-certs\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.370064 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.370047    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-data\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.370177 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.370163    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-podinfo\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.370304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.370288    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac04ed6c-cedf-460c-b512-a9a2380a9793-istiod-ca-cert\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.370516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.370499    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-socket\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.370697 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.370679    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-token\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.372810 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.372322    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-certs\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.372810 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.372720    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac04ed6c-cedf-460c-b512-a9a2380a9793-istiod-ca-cert\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.373053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.372997    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-socket\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.373279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.373242    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-credential-socket\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.374342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.374316    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-data\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.375344 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.375301    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-envoy\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.398996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.393651    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-podinfo\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.453569 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.453490    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-token\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.525068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.508666    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q2dj\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-kube-api-access-6q2dj\") pod \"e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") " pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.588961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.580590    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:34.724068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.722494    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" event={"ID":"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6","Type":"ContainerStarted","Data":"1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0"}
Nov 03 10:46:34.724068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.722563    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" event={"ID":"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6","Type":"ContainerStarted","Data":"5b15191b3f2fc0e41a9ed30f9eda70b65b8ca354877fbd76248632b1a6ce90fd"}
Nov 03 10:46:34.755343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:34.755280    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podStartSLOduration=2.755259747 podStartE2EDuration="2.755259747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:34.753256789 +0000 UTC m=+1038.658111232" watchObservedRunningTime="2026-11-03 10:46:34.755259747 +0000 UTC m=+1038.660114190"
Nov 03 10:46:35.103982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.103935    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"]
Nov 03 10:46:35.125032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.124484    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:35.125032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.124589    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:35.125032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.124632    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:35.220691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.220667    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:35.227402 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.227379    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:35.731460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.731426    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" event={"ID":"ac04ed6c-cedf-460c-b512-a9a2380a9793","Type":"ContainerStarted","Data":"156a5e1b670664db78fb3b2fb5a340bace8e8aa50b371ca142c17010f70633b6"}
Nov 03 10:46:35.731670 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.731489    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" event={"ID":"ac04ed6c-cedf-460c-b512-a9a2380a9793","Type":"ContainerStarted","Data":"d881197fb4160a1b4ea909caea4d92f2d8f6fa9d655b0f3cf6d992d54a3fc224"}
Nov 03 10:46:35.731736 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.731670    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:35.732802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.732782    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:46:35.753428 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:35.753381    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" podStartSLOduration=1.75336823 podStartE2EDuration="1.75336823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:35.752306405 +0000 UTC m=+1039.657160848" watchObservedRunningTime="2026-11-03 10:46:35.75336823 +0000 UTC m=+1039.658222663"
Nov 03 10:46:36.588096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:36.588063    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:36.594335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:36.594309    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:36.740261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:36.740228    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:36.741289 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:36.741267    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:46:43.190761 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:46:43.190715    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:46:44.460764 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.460729    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"]
Nov 03 10:46:44.467099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.466616    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.471335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.471310    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7d8392-serving-cert\""
Nov 03 10:46:44.472281 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.472255    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"]
Nov 03 10:46:44.600197 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.600153    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-metrics-certs\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.600197 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.600199    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbjhn\" (UniqueName: \"kubernetes.io/projected/4194a12b-2b39-413e-b820-40eaacc662df-kube-api-access-wbjhn\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.600450 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.600428    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4194a12b-2b39-413e-b820-40eaacc662df-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.600488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.600480    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-maas-api-tls\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.710626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.710587    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4194a12b-2b39-413e-b820-40eaacc662df-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.710981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.710903    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-maas-api-tls\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.711237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.711218    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-metrics-certs\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.711359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.711342    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbjhn\" (UniqueName: \"kubernetes.io/projected/4194a12b-2b39-413e-b820-40eaacc662df-kube-api-access-wbjhn\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.712761 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.712738    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4194a12b-2b39-413e-b820-40eaacc662df-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.722157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.722062    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-metrics-certs\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.750638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.750592    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbjhn\" (UniqueName: \"kubernetes.io/projected/4194a12b-2b39-413e-b820-40eaacc662df-kube-api-access-wbjhn\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.750898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.750869    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-maas-api-tls\") pod \"maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:44.791383 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:44.791349    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:45.234061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.231622    5393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:46:45.241337 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.240121    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"]
Nov 03 10:46:45.313133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.313100    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"]
Nov 03 10:46:45.319107 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.319073    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.336560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.336507    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"]
Nov 03 10:46:45.341766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.341732    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"]
Nov 03 10:46:45.341908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.341872    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.360102 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.354018    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"]
Nov 03 10:46:45.474885 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.474793    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77b64a-1da8-45e6-825c-81dee0a0366a-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.474885 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.474857    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475002    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475033    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475140    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475167    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475194    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475255    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475271    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-kube-api-access-9bzcs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475332    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwfk\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-kube-api-access-qzwfk\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.475387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475355    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.475669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475488    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.475669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475513    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.475669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475534    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3270be3d-f327-4ba8-862f-e88bdf0583f1-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.475669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475585    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.475669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475627    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.475669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475656    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.475842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.475704    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.576989    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577058    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577091    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577159    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577255    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77b64a-1da8-45e6-825c-81dee0a0366a-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577320    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577744    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577857    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577890    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.577978    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578014    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578054    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578091    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578114    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-kube-api-access-9bzcs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578174    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwfk\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-kube-api-access-qzwfk\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578207    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578269    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578313    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.578343    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3270be3d-f327-4ba8-862f-e88bdf0583f1-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.579159    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3270be3d-f327-4ba8-862f-e88bdf0583f1-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.579453    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.579715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.579669    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.581072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.580416    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.581072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.580627    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.581260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.581223    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77b64a-1da8-45e6-825c-81dee0a0366a-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.592115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.583435    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.592115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.583665    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.592115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.586439    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.594248 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.593762    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.594563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.594539    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.594683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.594646    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.595034 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.595013    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.595614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.595597    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwfk\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-kube-api-access-qzwfk\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.598182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.598158    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.598773 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.598750    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-kube-api-access-9bzcs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.599015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.598987    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-552v8\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.636040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.636006    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:45.668891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.668857    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:45.903450 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.903409    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7" event={"ID":"4194a12b-2b39-413e-b820-40eaacc662df","Type":"ContainerStarted","Data":"02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658"}
Nov 03 10:46:45.903633 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.903478    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7" event={"ID":"4194a12b-2b39-413e-b820-40eaacc662df","Type":"ContainerStarted","Data":"836a5db119422b32523d7a7aa51ad5cbaf95c40994fd0c0c155dc5866ff60d68"}
Nov 03 10:46:45.903746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.903725    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:45.928994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.928803    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7" podStartSLOduration=1.422815846 podStartE2EDuration="1.928788024s" totalImagesPullingTime="505.972178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:45.923556464 +0000 UTC m=+1049.828410906" watchObservedRunningTime="2026-11-03 10:46:45.928788024 +0000 UTC m=+1049.833642465"
Nov 03 10:46:45.932626 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:46:45.932593    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b77b64a_1da8_45e6_825c_81dee0a0366a.slice/crio-7aabc6e7c7e6f88fbc856bd90811e467aa1d80da0ac132c10a5dd85de05b3bd5 WatchSource:0}: Error finding container 7aabc6e7c7e6f88fbc856bd90811e467aa1d80da0ac132c10a5dd85de05b3bd5: Status 404 returned error can't find the container with id 7aabc6e7c7e6f88fbc856bd90811e467aa1d80da0ac132c10a5dd85de05b3bd5
Nov 03 10:46:45.934950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.934776    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"]
Nov 03 10:46:45.937274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.937246    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:45.937380 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.937309    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:45.937380 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.937340    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:45.980640 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.980614    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"]
Nov 03 10:46:45.981564 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:46:45.981533    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3270be3d_f327_4ba8_862f_e88bdf0583f1.slice/crio-c9f9eef2efd6dfaf7eabb6d103722e679ae970c60d12ec7e7eb45836723fb6ec WatchSource:0}: Error finding container c9f9eef2efd6dfaf7eabb6d103722e679ae970c60d12ec7e7eb45836723fb6ec: Status 404 returned error can't find the container with id c9f9eef2efd6dfaf7eabb6d103722e679ae970c60d12ec7e7eb45836723fb6ec
Nov 03 10:46:45.984749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.984712    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:45.984848 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.984826    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:45.984909 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:45.984870    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:46:46.975767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:46.975731    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" event={"ID":"3270be3d-f327-4ba8-862f-e88bdf0583f1","Type":"ContainerStarted","Data":"3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5"}
Nov 03 10:46:47.044666 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.044637    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" event={"ID":"3270be3d-f327-4ba8-862f-e88bdf0583f1","Type":"ContainerStarted","Data":"c9f9eef2efd6dfaf7eabb6d103722e679ae970c60d12ec7e7eb45836723fb6ec"}
Nov 03 10:46:47.044980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.044954    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" event={"ID":"8b77b64a-1da8-45e6-825c-81dee0a0366a","Type":"ContainerStarted","Data":"5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6"}
Nov 03 10:46:47.045156 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.045139    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" event={"ID":"8b77b64a-1da8-45e6-825c-81dee0a0366a","Type":"ContainerStarted","Data":"7aabc6e7c7e6f88fbc856bd90811e467aa1d80da0ac132c10a5dd85de05b3bd5"}
Nov 03 10:46:47.045281 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.044067    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podStartSLOduration=2.044043776 podStartE2EDuration="2.044043776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:47.011343109 +0000 UTC m=+1050.916197587" watchObservedRunningTime="2026-11-03 10:46:47.044043776 +0000 UTC m=+1050.948898231"
Nov 03 10:46:47.086379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.085566    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podStartSLOduration=2.085540965 podStartE2EDuration="2.085540965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:46:47.036624763 +0000 UTC m=+1050.941479255" watchObservedRunningTime="2026-11-03 10:46:47.085540965 +0000 UTC m=+1050.990395406"
Nov 03 10:46:47.658951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.644344    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:47.670954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.669409    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:47.731190 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.731155    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 10:46:47.731493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.731453    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 10:46:47.732091 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.732068    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-552v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:46:47.744033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:47.743993    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 10:46:48.654966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:48.643207    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-552v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:46:48.654966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:48.643265    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 10:46:48.701943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:48.698050    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 10:46:48.701943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:48.698112    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 10:46:49.638876 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:49.638367    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-552v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:46:49.638876 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:49.638427    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 10:46:49.673417 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:49.673378    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 10:46:49.674047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:49.674010    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 10:46:50.642944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:50.642498    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-552v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:46:50.642944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:50.642576    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 10:46:50.670474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:50.670431    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 10:46:50.670736 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:50.670505    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 10:46:51.638372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:51.638331    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-552v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:46:51.638991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:51.638954    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 10:46:51.669838 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:51.669790    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 10:46:51.670059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:51.669867    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 10:46:52.002140 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:52.002046    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:46:52.008765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:52.008736    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:46:52.640430 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:52.640392    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-552v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:46:52.641025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:52.640988    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 10:46:52.669580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:52.669543    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 10:46:52.669756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:52.669615    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 10:46:53.645945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:53.645347    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-552v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 10:46:53.645945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:53.645411    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 10:46:53.669711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:53.669665    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 10:46:53.669886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:53.669728    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 10:46:55.412951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:55.412868    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:55.433951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:55.415249    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:55.433951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:55.423173    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:46:55.498640 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:55.497989    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:55.500122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:55.500098    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:46:55.504526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:46:55.504508    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:47:00.241586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.241416    5393 patch_prober.go:29] interesting pod/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": read tcp 10.134.0.2:60696->10.134.0.83:15021: read: connection reset by peer" start-of-body=
Nov 03 10:47:00.241586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.241486    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": read tcp 10.134.0.2:60696->10.134.0.83:15021: read: connection reset by peer"
Nov 03 10:47:00.359780 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.354838    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"]
Nov 03 10:47:00.359780 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.359523    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.362147 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.361890    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"]
Nov 03 10:47:00.407551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.407521    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.407551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.407552    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.407820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.407571    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133865c0-b83e-4246-95ec-731579037dd5-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.407820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.407735    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133865c0-b83e-4246-95ec-731579037dd5-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.407820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.407797    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4c8\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-kube-api-access-pm4c8\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.408025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.407821    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.408025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.407891    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.408025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.408020    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.408187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.408059    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.509942    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4c8\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-kube-api-access-pm4c8\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.509996    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510060    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510200    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510263    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510372    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510388    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510512 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510404    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133865c0-b83e-4246-95ec-731579037dd5-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510512 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510486    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133865c0-b83e-4246-95ec-731579037dd5-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510521    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-data\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510605    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-certs\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510802    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.510824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.510815    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-credential-socket\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.511124 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.511072    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133865c0-b83e-4246-95ec-731579037dd5-istiod-ca-cert\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.512682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.512662    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-envoy\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.512976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.512959    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133865c0-b83e-4246-95ec-731579037dd5-istio-podinfo\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.517407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.517387    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-istio-token\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.517845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.517823    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4c8\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-kube-api-access-pm4c8\") pod \"e2e-mt-ab592834-openshift-default-7f97f8769-p75w4\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") " pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.684658 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.684627    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:00.972196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.972165    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"]
Nov 03 10:47:00.972318 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:47:00.972232    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133865c0_b83e_4246_95ec_731579037dd5.slice/crio-d172020948f75627da38c4a4aa9017ffbe1ceb42310e08593b24e8aa1cd77488 WatchSource:0}: Error finding container d172020948f75627da38c4a4aa9017ffbe1ceb42310e08593b24e8aa1cd77488: Status 404 returned error can't find the container with id d172020948f75627da38c4a4aa9017ffbe1ceb42310e08593b24e8aa1cd77488
Nov 03 10:47:00.975162 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.975119    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:00.975248 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.975211    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:00.975291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:00.975259    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:01.364133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.364097    5393 generic.go:356] "Generic (PLEG): container finished" podID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerID="10e36722a4b51331f119975123abe9cfaf46b32a01b4d32f06f017874f48500a" exitCode=0
Nov 03 10:47:01.364577 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.364168    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" event={"ID":"9a48dd6f-07bc-4633-aae0-54d49adc6bea","Type":"ContainerDied","Data":"10e36722a4b51331f119975123abe9cfaf46b32a01b4d32f06f017874f48500a"}
Nov 03 10:47:01.364686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.364667    5393 scope.go:122] "RemoveContainer" containerID="10e36722a4b51331f119975123abe9cfaf46b32a01b4d32f06f017874f48500a"
Nov 03 10:47:01.366201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.366178    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" event={"ID":"133865c0-b83e-4246-95ec-731579037dd5","Type":"ContainerStarted","Data":"7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46"}
Nov 03 10:47:01.366295 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.366227    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" event={"ID":"133865c0-b83e-4246-95ec-731579037dd5","Type":"ContainerStarted","Data":"d172020948f75627da38c4a4aa9017ffbe1ceb42310e08593b24e8aa1cd77488"}
Nov 03 10:47:01.366437 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.366417    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:01.366490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.366478    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:01.366524 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.366514    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:01.697003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:01.690017    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:02.375772 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:02.375736    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" event={"ID":"9a48dd6f-07bc-4633-aae0-54d49adc6bea","Type":"ContainerStarted","Data":"ac697e4b09e310468d87eddd636e884b6c6cd7ccf9d36e663b75c229bca50932"}
Nov 03 10:47:02.404945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:02.402902    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podStartSLOduration=2.402880267 podStartE2EDuration="2.402880267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:01.411467387 +0000 UTC m=+1065.316321838" watchObservedRunningTime="2026-11-03 10:47:02.402880267 +0000 UTC m=+1066.307734710"
Nov 03 10:47:02.689003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:02.687904    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 03 10:47:02.689003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:02.688036    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 03 10:47:03.340797 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:03.340766    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:47:03.341178 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:03.341152    5393 patch_prober.go:29] interesting pod/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:47:03.341248 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:03.341206    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 10:47:03.724740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:03.685354    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 03 10:47:03.724740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:03.724186    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 03 10:47:04.341331 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:04.341296    5393 patch_prober.go:29] interesting pod/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:47:04.341790 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:04.341750    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 10:47:04.685949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:04.685896    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 03 10:47:04.686231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:04.686205    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 03 10:47:05.346946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:05.343170    5393 patch_prober.go:29] interesting pod/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:47:05.346946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:05.343234    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 10:47:05.692000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:05.691358    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 03 10:47:05.692000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:05.691421    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 03 10:47:06.340340 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:06.340303    5393 patch_prober.go:29] interesting pod/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:47:06.340519 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:06.340362    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 10:47:06.685322 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:06.685283    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 03 10:47:06.685943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:06.685891    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 03 10:47:07.341944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:07.341632    5393 patch_prober.go:29] interesting pod/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 10:47:07.341944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:07.341688    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 10:47:07.693953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:07.691332    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 03 10:47:07.693953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:07.691393    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 03 10:47:08.456506 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:08.456417    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:47:08.457054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:08.457019    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:47:08.458379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:08.458356    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:47:08.691239 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:08.691208    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:08.691676 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:08.691654    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:08.692446 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:08.692424    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:47:13.236536 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:47:13.236495    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:47:16.439615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:16.439588    5393 scope.go:122] "RemoveContainer" containerID="e07e91b94fb8b4a207fc7eea671ef4de1c31563d3ed008b2a6692f8e5bcb840f"
Nov 03 10:47:27.046944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.028420    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-2f99f8"]
Nov 03 10:47:27.143617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.143575    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-2f99f8"]
Nov 03 10:47:27.143983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.143797    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-2f99f8"
Nov 03 10:47:27.154236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.154191    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5zm6k\""
Nov 03 10:47:27.307034 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.297116    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5h6\" (UniqueName: \"kubernetes.io/projected/7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a-kube-api-access-hx5h6\") pod \"test-curl-7603-2f99f8\" (UID: \"7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a\") " pod="opendatahub/test-curl-7603-2f99f8"
Nov 03 10:47:27.410267 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.410231    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5h6\" (UniqueName: \"kubernetes.io/projected/7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a-kube-api-access-hx5h6\") pod \"test-curl-7603-2f99f8\" (UID: \"7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a\") " pod="opendatahub/test-curl-7603-2f99f8"
Nov 03 10:47:27.418472 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.418447    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5h6\" (UniqueName: \"kubernetes.io/projected/7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a-kube-api-access-hx5h6\") pod \"test-curl-7603-2f99f8\" (UID: \"7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a\") " pod="opendatahub/test-curl-7603-2f99f8"
Nov 03 10:47:27.477244 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.477211    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-2f99f8"
Nov 03 10:47:27.764067 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:47:27.764031    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7054ba8d_7cfd_4e2a_8bc3_907b2c446d8a.slice/crio-c8154464f351f141e405529b37f09692ab9d2e0a995282b2f1de279a1b84c7dd WatchSource:0}: Error finding container c8154464f351f141e405529b37f09692ab9d2e0a995282b2f1de279a1b84c7dd: Status 404 returned error can't find the container with id c8154464f351f141e405529b37f09692ab9d2e0a995282b2f1de279a1b84c7dd
Nov 03 10:47:27.764226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.764185    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-2f99f8"]
Nov 03 10:47:27.841223 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:27.841178    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-2f99f8" event={"ID":"7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a","Type":"ContainerStarted","Data":"c8154464f351f141e405529b37f09692ab9d2e0a995282b2f1de279a1b84c7dd"}
Nov 03 10:47:28.849290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:28.849259    5393 generic.go:356] "Generic (PLEG): container finished" podID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerID="156a5e1b670664db78fb3b2fb5a340bace8e8aa50b371ca142c17010f70633b6" exitCode=0
Nov 03 10:47:28.850026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:28.849332    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" event={"ID":"ac04ed6c-cedf-460c-b512-a9a2380a9793","Type":"ContainerDied","Data":"156a5e1b670664db78fb3b2fb5a340bace8e8aa50b371ca142c17010f70633b6"}
Nov 03 10:47:28.850026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:28.849816    5393 scope.go:122] "RemoveContainer" containerID="156a5e1b670664db78fb3b2fb5a340bace8e8aa50b371ca142c17010f70633b6"
Nov 03 10:47:28.851494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:28.851462    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:28.851601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:28.851537    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:28.851601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:28.851569    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:47:29.861121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:29.861085    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" event={"ID":"ac04ed6c-cedf-460c-b512-a9a2380a9793","Type":"ContainerStarted","Data":"a4e70440a5c7d1645c09bf9b4fb1243782d0ca22481873dc94ff42e6265c2d54"}
Nov 03 10:47:29.948789 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:29.942046    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"]
Nov 03 10:47:29.948789 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:29.942278    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy" containerID="cri-o://5c339e369dce91b4570798d5558560efe06aaf9d8bc1b3700c17b8cef5560bef" gracePeriod=30
Nov 03 10:47:30.597979 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:30.589737    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:47:30.597979 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:30.589880    5393 patch_prober.go:29] interesting pod/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 10:47:30.597979 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:30.589951    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 10:47:31.606945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:31.605329    5393 patch_prober.go:29] interesting pod/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 10:47:31.606945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:31.605392    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 10:47:32.588998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:32.588339    5393 patch_prober.go:29] interesting pod/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 10:47:32.588998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:32.588402    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 10:47:32.910698 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:32.910660    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-2f99f8" event={"ID":"7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a","Type":"ContainerStarted","Data":"5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115"}
Nov 03 10:47:32.928868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:32.928818    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7603-2f99f8" podStartSLOduration=2.405164353 podStartE2EDuration="6.928804979s" totalImagesPullingTime="4.523640626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:32.926738403 +0000 UTC m=+1096.831592845" watchObservedRunningTime="2026-11-03 10:47:32.928804979 +0000 UTC m=+1096.833659419"
Nov 03 10:47:33.022829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:33.022791    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"]
Nov 03 10:47:33.026629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:33.026598    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d8feaf-npm9s"]
Nov 03 10:47:33.605945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:33.598349    5393 patch_prober.go:29] interesting pod/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 10:47:33.605945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:33.598415    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 10:47:33.818942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:33.817584    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-2f99f8"]
Nov 03 10:47:33.823855 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:33.823809    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-2f99f8"]
Nov 03 10:47:34.231583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:34.231548    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="807c2e3e-ede1-4a38-a8f5-7c6e90eb8846" path="/var/lib/kubelet/pods/807c2e3e-ede1-4a38-a8f5-7c6e90eb8846/volumes"
Nov 03 10:47:34.599002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:34.588330    5393 patch_prober.go:29] interesting pod/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 10:47:34.599002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:34.588408    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 10:47:34.927592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:34.927477    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7603-2f99f8" podUID="7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a" containerName="test-curl-7603-2f99f8" containerID="cri-o://5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115" gracePeriod=2
Nov 03 10:47:35.596190 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.596167    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:47:35.597119 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.597096    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:47:35.597649 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.597633    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:47:35.941834 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.941800    5393 generic.go:356] "Generic (PLEG): container finished" podID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerID="5c339e369dce91b4570798d5558560efe06aaf9d8bc1b3700c17b8cef5560bef" exitCode=0
Nov 03 10:47:35.942014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.941853    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" event={"ID":"4103a55e-db0d-4d1f-9c9b-1e49d1102c65","Type":"ContainerDied","Data":"5c339e369dce91b4570798d5558560efe06aaf9d8bc1b3700c17b8cef5560bef"}
Nov 03 10:47:35.942014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.941908    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm" event={"ID":"4103a55e-db0d-4d1f-9c9b-1e49d1102c65","Type":"ContainerDied","Data":"eaf9a3af4dbc4f679478a0ffd3cc36bdf23f675627ab776cf9729343238d4f0f"}
Nov 03 10:47:35.942014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.941980    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaf9a3af4dbc4f679478a0ffd3cc36bdf23f675627ab776cf9729343238d4f0f"
Nov 03 10:47:35.942014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.941947    5393 scope.go:122] "RemoveContainer" containerID="c961e6abb66d2f3df226c323441930fbd22fc8a8513d349e27d24faae8f2f741"
Nov 03 10:47:35.952378 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:35.952353    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:47:36.051240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051208    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-kube-api-access-vv88b\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-kube-api-access-vv88b\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.051466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051284    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-envoy\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.051466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051338    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-token\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-token\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.051466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051372    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-socket\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.051466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051445    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-certs\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.051754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051707    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istiod-ca-cert\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.051754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051732    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-socket" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:36.051907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051775    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-certs" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:36.051907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051793    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-podinfo\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.051907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051822    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-credential-socket\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.051907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.051856    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-data\") pod \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\" (UID: \"4103a55e-db0d-4d1f-9c9b-1e49d1102c65\") "
Nov 03 10:47:36.052316 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.052108    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-credential-socket" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:36.052316 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.052130    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istiod-ca-cert" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:36.052435 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.052358    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-data" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:36.053196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.053174    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.053317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.053198    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.053317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.053212    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.053317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.053226    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.053317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.053239    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.054023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.053997    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-kube-api-access-vv88b" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "kube-api-access-vv88b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:36.054123 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.054062    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-envoy" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:47:36.054123 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.054083    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-token" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:36.054246 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.054232    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-podinfo" pod "4103a55e-db0d-4d1f-9c9b-1e49d1102c65" (UID: "4103a55e-db0d-4d1f-9c9b-1e49d1102c65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:47:36.154426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.154394    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.154426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.154423    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv88b\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-kube-api-access-vv88b\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.154643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.154437    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.154643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.154445    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4103a55e-db0d-4d1f-9c9b-1e49d1102c65-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:36.951641 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:36.951616    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"
Nov 03 10:47:37.012722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.012689    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"]
Nov 03 10:47:37.017911 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.017882    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-141053-openshift-default-6694f679f7-2vzmm"]
Nov 03 10:47:37.154114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.154088    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-2f99f8"
Nov 03 10:47:37.286059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.285971    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a-kube-api-access-hx5h6\" (UniqueName: \"kubernetes.io/projected/7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a-kube-api-access-hx5h6\") pod \"7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a\" (UID: \"7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a\") "
Nov 03 10:47:37.288329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.288301    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a-kube-api-access-hx5h6" pod "7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a" (UID: "7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a"). InnerVolumeSpecName "kube-api-access-hx5h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:37.398678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.397846    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx5h6\" (UniqueName: \"kubernetes.io/projected/7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a-kube-api-access-hx5h6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:37.960475 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.960432    5393 generic.go:356] "Generic (PLEG): container finished" podID="7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a" containerID="5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115" exitCode=137
Nov 03 10:47:37.960901 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.960507    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-2f99f8"
Nov 03 10:47:37.960901 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.960520    5393 scope.go:122] "RemoveContainer" containerID="5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115"
Nov 03 10:47:37.971661 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.971644    5393 scope.go:122] "RemoveContainer" containerID="5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115"
Nov 03 10:47:37.971991 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:47:37.971968    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115\": container with ID starting with 5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115 not found: ID does not exist" containerID="5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115"
Nov 03 10:47:37.972097 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:47:37.971987    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7603-2f99f8\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a" pod="opendatahub/test-curl-7603-2f99f8"
Nov 03 10:47:37.972097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:37.972000    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115"} err="failed to get container status \"5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115\": rpc error: code = NotFound desc = could not find container \"5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115\": container with ID starting with 5f54eea333d903a75db41a56d7ff0bc36d5bdee591d27611cdf44cc4955e0115 not found: ID does not exist"
Nov 03 10:47:38.230730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:38.230650    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" path="/var/lib/kubelet/pods/4103a55e-db0d-4d1f-9c9b-1e49d1102c65/volumes"
Nov 03 10:47:38.231114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:38.231100    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a" path="/var/lib/kubelet/pods/7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a/volumes"
Nov 03 10:47:43.190498 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:47:43.190461    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:47:48.432414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.432377    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"]
Nov 03 10:47:48.433465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.433436    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a" containerName="test-curl-7603-2f99f8"
Nov 03 10:47:48.433589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.433500    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a" containerName="test-curl-7603-2f99f8"
Nov 03 10:47:48.433589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.433517    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy"
Nov 03 10:47:48.433589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.433524    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy"
Nov 03 10:47:48.433731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.433698    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy"
Nov 03 10:47:48.433731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.433726    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7054ba8d-7cfd-4e2a-8bc3-907b2c446d8a" containerName="test-curl-7603-2f99f8"
Nov 03 10:47:48.440079 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.440051    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.451297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.451261    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:47:48.459949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.452505    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.460293 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.460244    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.460596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.460568    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:47:48.475941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.460888    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6sp\" (UniqueName: \"kubernetes.io/projected/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-kube-api-access-rb6sp\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.493859 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.486016    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"]
Nov 03 10:47:48.579028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.578992    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.579210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.579152    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6sp\" (UniqueName: \"kubernetes.io/projected/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-kube-api-access-rb6sp\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.579404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.579388    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.579768 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.579746    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.582043 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.582017    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.588444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.588418    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6sp\" (UniqueName: \"kubernetes.io/projected/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-kube-api-access-rb6sp\") pod \"maas-api-revoke-keys-e2e-api-a-7d8392-twnkg\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:48.804302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:48.804203    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:49.086366 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:49.086331    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"]
Nov 03 10:47:49.087193 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:47:49.087167    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e6b4d1_d3da_419f_8a4b_0b99808fa0cc.slice/crio-e8a33e21d60eb2cc4588163ecddcc2cdf2374e6d12592d52ac57acf7267afc94 WatchSource:0}: Error finding container e8a33e21d60eb2cc4588163ecddcc2cdf2374e6d12592d52ac57acf7267afc94: Status 404 returned error can't find the container with id e8a33e21d60eb2cc4588163ecddcc2cdf2374e6d12592d52ac57acf7267afc94
Nov 03 10:47:49.153363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:49.153320    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg" event={"ID":"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc","Type":"ContainerStarted","Data":"e8a33e21d60eb2cc4588163ecddcc2cdf2374e6d12592d52ac57acf7267afc94"}
Nov 03 10:47:50.193147 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:50.193081    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg" event={"ID":"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc","Type":"ContainerStarted","Data":"00be0a73d5eac571ba84c4bd8c7cb765924f10235a02806f83a226e21b1c9487"}
Nov 03 10:47:50.251946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:50.247674    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg" podStartSLOduration=2.247652551 podStartE2EDuration="2.247652551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:47:50.221146698 +0000 UTC m=+1114.126001140" watchObservedRunningTime="2026-11-03 10:47:50.247652551 +0000 UTC m=+1114.152506992"
Nov 03 10:47:51.201278 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:51.201242    5393 generic.go:356] "Generic (PLEG): container finished" podID="87e6b4d1-d3da-419f-8a4b-0b99808fa0cc" containerID="00be0a73d5eac571ba84c4bd8c7cb765924f10235a02806f83a226e21b1c9487" exitCode=0
Nov 03 10:47:51.201766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:51.201495    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg" event={"ID":"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc","Type":"ContainerDied","Data":"00be0a73d5eac571ba84c4bd8c7cb765924f10235a02806f83a226e21b1c9487"}
Nov 03 10:47:52.468247 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.468187    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:52.581592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.581557    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-xks-ca\") pod \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") "
Nov 03 10:47:52.581767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.581672    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-kube-api-access-rb6sp\" (UniqueName: \"kubernetes.io/projected/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-kube-api-access-rb6sp\") pod \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") "
Nov 03 10:47:52.581815 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.581785    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-openshift-ca\") pod \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\" (UID: \"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc\") "
Nov 03 10:47:52.582219 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.582184    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-openshift-ca" pod "87e6b4d1-d3da-419f-8a4b-0b99808fa0cc" (UID: "87e6b4d1-d3da-419f-8a4b-0b99808fa0cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:47:52.582443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.582428    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.583757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.583733    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-kube-api-access-rb6sp" pod "87e6b4d1-d3da-419f-8a4b-0b99808fa0cc" (UID: "87e6b4d1-d3da-419f-8a4b-0b99808fa0cc"). InnerVolumeSpecName "kube-api-access-rb6sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:47:52.583831 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.583789    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-xks-ca" pod "87e6b4d1-d3da-419f-8a4b-0b99808fa0cc" (UID: "87e6b4d1-d3da-419f-8a4b-0b99808fa0cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:47:52.684443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.684411    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6sp\" (UniqueName: \"kubernetes.io/projected/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-kube-api-access-rb6sp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:52.684443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:52.684439    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:47:53.217533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:53.217501    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"
Nov 03 10:47:53.217533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:53.217500    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg" event={"ID":"87e6b4d1-d3da-419f-8a4b-0b99808fa0cc","Type":"ContainerDied","Data":"e8a33e21d60eb2cc4588163ecddcc2cdf2374e6d12592d52ac57acf7267afc94"}
Nov 03 10:47:53.217756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:53.217594    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8a33e21d60eb2cc4588163ecddcc2cdf2374e6d12592d52ac57acf7267afc94"
Nov 03 10:47:57.312498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:57.312422    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"]
Nov 03 10:47:57.313210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:47:57.313164    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy" containerID="cri-o://ac697e4b09e310468d87eddd636e884b6c6cd7ccf9d36e663b75c229bca50932" gracePeriod=30
Nov 03 10:48:02.458114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.458082    5393 generic.go:356] "Generic (PLEG): container finished" podID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerID="ac697e4b09e310468d87eddd636e884b6c6cd7ccf9d36e663b75c229bca50932" exitCode=0
Nov 03 10:48:02.458494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.458167    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" event={"ID":"9a48dd6f-07bc-4633-aae0-54d49adc6bea","Type":"ContainerDied","Data":"ac697e4b09e310468d87eddd636e884b6c6cd7ccf9d36e663b75c229bca50932"}
Nov 03 10:48:02.458494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.458250    5393 scope.go:122] "RemoveContainer" containerID="10e36722a4b51331f119975123abe9cfaf46b32a01b4d32f06f017874f48500a"
Nov 03 10:48:02.661202 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.661173    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:48:02.671559 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671524    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-credential-socket\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.671729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671567    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-token\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-token\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.671729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671607    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-socket\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.671729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671681    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-envoy\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.671729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671702    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-data\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.672016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671747    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istiod-ca-cert\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.672016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671786    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-kube-api-access-jlc64\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-kube-api-access-jlc64\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.672016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671834    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-certs\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.672016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671855    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-credential-socket" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:02.672016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.671879    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-podinfo\") pod \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\" (UID: \"9a48dd6f-07bc-4633-aae0-54d49adc6bea\") "
Nov 03 10:48:02.672016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.672007    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-data" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:02.672341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.672021    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-socket" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:02.672341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.672293    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istiod-ca-cert" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:48:02.673125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.672895    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-certs" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:02.673125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.673002    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:02.673125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.673057    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:02.673125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.673073    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:02.673125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.673085    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:02.674313 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.674288    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-podinfo" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:48:02.674313 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.674297    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-envoy" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:02.674604 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.674580    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-token" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:02.674644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.674617    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-kube-api-access-jlc64" pod "9a48dd6f-07bc-4633-aae0-54d49adc6bea" (UID: "9a48dd6f-07bc-4633-aae0-54d49adc6bea"). InnerVolumeSpecName "kube-api-access-jlc64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:02.821941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.816277    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:02.821941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.816321    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlc64\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-kube-api-access-jlc64\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:02.821941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.816338    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a48dd6f-07bc-4633-aae0-54d49adc6bea-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:02.821941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.816352    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:02.821941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:02.816365    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a48dd6f-07bc-4633-aae0-54d49adc6bea-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:03.600943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:03.599843    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw" event={"ID":"9a48dd6f-07bc-4633-aae0-54d49adc6bea","Type":"ContainerDied","Data":"4312b5b57628e70a2998e463bb116b48fd2974ae75997b64916772b17a4f1aee"}
Nov 03 10:48:03.600943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:03.599935    5393 scope.go:122] "RemoveContainer" containerID="ac697e4b09e310468d87eddd636e884b6c6cd7ccf9d36e663b75c229bca50932"
Nov 03 10:48:03.600943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:03.600049    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"
Nov 03 10:48:03.638515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:03.638483    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"]
Nov 03 10:48:03.641957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:03.641931    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a917b1-openshift-default-7bb7654c89-dqlhw"]
Nov 03 10:48:04.235827 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:04.235785    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" path="/var/lib/kubelet/pods/9a48dd6f-07bc-4633-aae0-54d49adc6bea/volumes"
Nov 03 10:48:06.774093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:06.774051    5393 patch_prober.go:29] interesting pod/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:48:06.774571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:06.774113    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:48:10.234426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.222198    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"]
Nov 03 10:48:10.234426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.224000    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7" podUID="4194a12b-2b39-413e-b820-40eaacc662df" containerName="maas-api" containerID="cri-o://02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658" gracePeriod=30
Nov 03 10:48:10.573003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.571333    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"]
Nov 03 10:48:10.636993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.620398    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:48:10.637363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.610013    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d8392-twnkg"]
Nov 03 10:48:10.775940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.772556    5393 generic.go:356] "Generic (PLEG): container finished" podID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerID="1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0" exitCode=0
Nov 03 10:48:10.775940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.772897    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" event={"ID":"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6","Type":"ContainerDied","Data":"1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0"}
Nov 03 10:48:10.775940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.773594    5393 scope.go:122] "RemoveContainer" containerID="1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0"
Nov 03 10:48:10.840001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.798420    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-maas-api-tls\") pod \"4194a12b-2b39-413e-b820-40eaacc662df\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") "
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.840387    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-metrics-certs\") pod \"4194a12b-2b39-413e-b820-40eaacc662df\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") "
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.840439    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4194a12b-2b39-413e-b820-40eaacc662df-kube-api-access-wbjhn\" (UniqueName: \"kubernetes.io/projected/4194a12b-2b39-413e-b820-40eaacc662df-kube-api-access-wbjhn\") pod \"4194a12b-2b39-413e-b820-40eaacc662df\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") "
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.840597    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4194a12b-2b39-413e-b820-40eaacc662df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4194a12b-2b39-413e-b820-40eaacc662df-openshift-service-ca\") pod \"4194a12b-2b39-413e-b820-40eaacc662df\" (UID: \"4194a12b-2b39-413e-b820-40eaacc662df\") "
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.845816    5393 generic.go:356] "Generic (PLEG): container finished" podID="4194a12b-2b39-413e-b820-40eaacc662df" containerID="02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658" exitCode=0
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.845996    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7" event={"ID":"4194a12b-2b39-413e-b820-40eaacc662df","Type":"ContainerDied","Data":"02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658"}
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.846066    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7" event={"ID":"4194a12b-2b39-413e-b820-40eaacc662df","Type":"ContainerDied","Data":"836a5db119422b32523d7a7aa51ad5cbaf95c40994fd0c0c155dc5866ff60d68"}
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.846096    5393 scope.go:122] "RemoveContainer" containerID="02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658"
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.846296    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"
Nov 03 10:48:10.868155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.864871    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-metrics-certs" pod "4194a12b-2b39-413e-b820-40eaacc662df" (UID: "4194a12b-2b39-413e-b820-40eaacc662df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:48:10.872816 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.872784    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:10.873070 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.873049    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:10.873218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.873202    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:10.886319 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.881251    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4194a12b-2b39-413e-b820-40eaacc662df-openshift-service-ca" pod "4194a12b-2b39-413e-b820-40eaacc662df" (UID: "4194a12b-2b39-413e-b820-40eaacc662df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:48:10.960248 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.960212    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4194a12b-2b39-413e-b820-40eaacc662df-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:10.960468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.960453    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:10.997589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:10.997529    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-maas-api-tls" pod "4194a12b-2b39-413e-b820-40eaacc662df" (UID: "4194a12b-2b39-413e-b820-40eaacc662df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:48:11.029498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:11.029443    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4194a12b-2b39-413e-b820-40eaacc662df-kube-api-access-wbjhn" pod "4194a12b-2b39-413e-b820-40eaacc662df" (UID: "4194a12b-2b39-413e-b820-40eaacc662df"). InnerVolumeSpecName "kube-api-access-wbjhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:11.042850 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:11.042824    5393 scope.go:122] "RemoveContainer" containerID="02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658"
Nov 03 10:48:11.044409 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:48:11.044380    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658\": container with ID starting with 02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658 not found: ID does not exist" containerID="02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658"
Nov 03 10:48:11.044516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:11.044417    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658"} err="failed to get container status \"02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658\": rpc error: code = NotFound desc = could not find container \"02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658\": container with ID starting with 02fcbc5184d5ba8067ad6e669c929ca8ecd13ea1eaa829c454419e325e0c1658 not found: ID does not exist"
Nov 03 10:48:11.062735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:11.062701    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbjhn\" (UniqueName: \"kubernetes.io/projected/4194a12b-2b39-413e-b820-40eaacc662df-kube-api-access-wbjhn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:11.062876 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:11.062737    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4194a12b-2b39-413e-b820-40eaacc662df-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:11.173661 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:11.173626    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"]
Nov 03 10:48:11.177198 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:11.177172    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d8392-5d658c5fcb-6rwq7"]
Nov 03 10:48:11.855337 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:11.855302    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" event={"ID":"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6","Type":"ContainerStarted","Data":"09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c"}
Nov 03 10:48:12.222435 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:12.222334    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:48:12.222614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:12.222462    5393 patch_prober.go:29] interesting pod/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:48:12.222614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:12.222503    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 10:48:12.232437 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:12.232407    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4194a12b-2b39-413e-b820-40eaacc662df" path="/var/lib/kubelet/pods/4194a12b-2b39-413e-b820-40eaacc662df/volumes"
Nov 03 10:48:12.232931 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:12.232901    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e6b4d1-d3da-419f-8a4b-0b99808fa0cc" path="/var/lib/kubelet/pods/87e6b4d1-d3da-419f-8a4b-0b99808fa0cc/volumes"
Nov 03 10:48:13.214407 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:48:13.214373    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:48:13.220884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:13.220857    5393 patch_prober.go:29] interesting pod/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:48:13.221056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:13.220909    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 10:48:14.220533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:14.220502    5393 patch_prober.go:29] interesting pod/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:48:14.220950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:14.220553    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 10:48:15.226947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:15.226474    5393 patch_prober.go:29] interesting pod/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:48:15.226947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:15.226553    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 10:48:16.220785 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:16.220749    5393 patch_prober.go:29] interesting pod/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:48:16.220995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:16.220804    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 10:48:17.229942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:17.225562    5393 patch_prober.go:29] interesting pod/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 10:48:17.229942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:17.225622    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 10:48:17.229942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:17.228578    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"]
Nov 03 10:48:17.229942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:17.228786    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy" containerID="cri-o://09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c" gracePeriod=30
Nov 03 10:48:18.861196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:18.859714    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"]
Nov 03 10:48:18.861196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:18.860074    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy" containerID="cri-o://a4e70440a5c7d1645c09bf9b4fb1243782d0ca22481873dc94ff42e6265c2d54" gracePeriod=30
Nov 03 10:48:20.298300 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298267    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"]
Nov 03 10:48:20.298852 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298839    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4194a12b-2b39-413e-b820-40eaacc662df" containerName="maas-api"
Nov 03 10:48:20.298895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298853    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4194a12b-2b39-413e-b820-40eaacc662df" containerName="maas-api"
Nov 03 10:48:20.298895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298869    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy"
Nov 03 10:48:20.298895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298875    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy"
Nov 03 10:48:20.299015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298905    5393 cpu_manager.go:414] "containerMap: removing container" podUID="87e6b4d1-d3da-419f-8a4b-0b99808fa0cc" containerName="revoke-keys"
Nov 03 10:48:20.299015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298911    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e6b4d1-d3da-419f-8a4b-0b99808fa0cc" containerName="revoke-keys"
Nov 03 10:48:20.299015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298945    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy"
Nov 03 10:48:20.299015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.298950    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy"
Nov 03 10:48:20.299133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.299039    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4194a12b-2b39-413e-b820-40eaacc662df" containerName="maas-api"
Nov 03 10:48:20.299133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.299049    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy"
Nov 03 10:48:20.299133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.299057    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy"
Nov 03 10:48:20.299133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.299063    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e6b4d1-d3da-419f-8a4b-0b99808fa0cc" containerName="revoke-keys"
Nov 03 10:48:20.299133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.299071    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4103a55e-db0d-4d1f-9c9b-1e49d1102c65" containerName="istio-proxy"
Nov 03 10:48:20.304004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.303986    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.323896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.323865    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"]
Nov 03 10:48:20.324875 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.324854    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-90bad4-openshift-default-dockercfg-jvg7r\""
Nov 03 10:48:20.372513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.372456    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-socket\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.372696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.372522    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-envoy\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.372696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.372651    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-podinfo\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.372910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.372751    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn56f\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-kube-api-access-wn56f\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.372910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.372790    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-certs\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.372910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.372832    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-credential-socket\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.373099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.372971    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-data\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.373099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.373007    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-token\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.373099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.373034    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4380b-8e6e-4505-a07e-4bf980ec7296-istiod-ca-cert\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.493003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.492962    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-data\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.493284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.493263    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-token\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.493410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.493395    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4380b-8e6e-4505-a07e-4bf980ec7296-istiod-ca-cert\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.493620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.493605    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-socket\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.493743 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.493723    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-envoy\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.493886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.493870    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-podinfo\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.494095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.494075    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn56f\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-kube-api-access-wn56f\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.494320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.494264    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-certs\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.494535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.494514    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-credential-socket\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.511229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.510618    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-socket\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.511762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.511729    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-credential-socket\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.513267 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.513214    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-data\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.514366 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.514317    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-certs\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.515226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.515149    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4380b-8e6e-4505-a07e-4bf980ec7296-istiod-ca-cert\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.538967 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.537820    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn56f\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-kube-api-access-wn56f\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.539424 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.539397    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-envoy\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.548986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.540580    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-podinfo\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.549588 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.549560    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-token\") pod \"e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") " pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.621944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.620036    5393 patch_prober.go:29] interesting pod/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:48:20.621944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.620111    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:48:20.621944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.620590    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:20.741530 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:20.741366    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:48:21.098535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.098507    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"]
Nov 03 10:48:21.099523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.099501    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy"
Nov 03 10:48:21.099615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.099526    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a48dd6f-07bc-4633-aae0-54d49adc6bea" containerName="istio-proxy"
Nov 03 10:48:21.124167 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.124134    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"]
Nov 03 10:48:21.124342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.124272    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.131583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.131534    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:48:21.131583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.131559    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:48:21.162295 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.162251    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k4x6\" (UniqueName: \"kubernetes.io/projected/23750ab7-5579-47ce-9168-3e02effb1e3d-kube-api-access-2k4x6\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.162490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.162472    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.162576 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.162561    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.264721 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.264683    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.264956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.264846    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4x6\" (UniqueName: \"kubernetes.io/projected/23750ab7-5579-47ce-9168-3e02effb1e3d-kube-api-access-2k4x6\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.265093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.265070    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.265677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.265649    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.267341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.267319    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.273469 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.273447    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4x6\" (UniqueName: \"kubernetes.io/projected/23750ab7-5579-47ce-9168-3e02effb1e3d-kube-api-access-2k4x6\") pod \"maas-api-revoke-sub-a1a9a12f-kkd6n\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.320966 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:48:21.320907    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d4380b_8e6e_4505_a07e_4bf980ec7296.slice/crio-e111c03c45de25753f79321a87cc6c2fb3d51b266b9e7d514494d46699890989 WatchSource:0}: Error finding container e111c03c45de25753f79321a87cc6c2fb3d51b266b9e7d514494d46699890989: Status 404 returned error can't find the container with id e111c03c45de25753f79321a87cc6c2fb3d51b266b9e7d514494d46699890989
Nov 03 10:48:21.321389 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.321095    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"]
Nov 03 10:48:21.323668 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.323635    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:21.323750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.323703    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:21.323750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.323731    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:21.468952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.468467    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:21.862770 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:48:21.862731    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23750ab7_5579_47ce_9168_3e02effb1e3d.slice/crio-721ebd3aa847c55e1ca07b8f3d6bd86ad16f0297eb6b8fa469c52aba35a47953 WatchSource:0}: Error finding container 721ebd3aa847c55e1ca07b8f3d6bd86ad16f0297eb6b8fa469c52aba35a47953: Status 404 returned error can't find the container with id 721ebd3aa847c55e1ca07b8f3d6bd86ad16f0297eb6b8fa469c52aba35a47953
Nov 03 10:48:21.864154 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:21.864124    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"]
Nov 03 10:48:22.090001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:22.089969    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" event={"ID":"28d4380b-8e6e-4505-a07e-4bf980ec7296","Type":"ContainerStarted","Data":"ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e"}
Nov 03 10:48:22.090157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:22.090022    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" event={"ID":"28d4380b-8e6e-4505-a07e-4bf980ec7296","Type":"ContainerStarted","Data":"e111c03c45de25753f79321a87cc6c2fb3d51b266b9e7d514494d46699890989"}
Nov 03 10:48:22.091357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:22.091335    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n" event={"ID":"23750ab7-5579-47ce-9168-3e02effb1e3d","Type":"ContainerStarted","Data":"721ebd3aa847c55e1ca07b8f3d6bd86ad16f0297eb6b8fa469c52aba35a47953"}
Nov 03 10:48:22.112167 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:22.112064    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podStartSLOduration=2.112049294 podStartE2EDuration="2.112049294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:22.109187632 +0000 UTC m=+1146.014042074" watchObservedRunningTime="2026-11-03 10:48:22.112049294 +0000 UTC m=+1146.016903734"
Nov 03 10:48:22.631484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:22.631400    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:22.637422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:22.637397    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:23.099302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.099262    5393 generic.go:356] "Generic (PLEG): container finished" podID="23750ab7-5579-47ce-9168-3e02effb1e3d" containerID="ae45d377f6407a332cf0c90eef157fb2751b6d34e7f22b0bd1c78509d25622f7" exitCode=0
Nov 03 10:48:23.099486 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.099300    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n" event={"ID":"23750ab7-5579-47ce-9168-3e02effb1e3d","Type":"ContainerDied","Data":"ae45d377f6407a332cf0c90eef157fb2751b6d34e7f22b0bd1c78509d25622f7"}
Nov 03 10:48:23.099878 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.099850    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:23.100954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.100903    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:48:23.548016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.547988    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:48:23.640601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.640560    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-kube-api-access-7mxtf\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-kube-api-access-7mxtf\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.641089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.640623    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-socket\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.641089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.640778    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-podinfo\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.641089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.640807    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-envoy\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.641089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.640877    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-certs\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.641089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.641025    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-credential-socket\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.641089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.641083    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istiod-ca-cert\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.641398 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.641301    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-socket" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:23.641491 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.641466    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-certs" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:23.641633 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.641608    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istiod-ca-cert" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:48:23.641868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.641818    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-token\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-token\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.641868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.641789    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-credential-socket" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:23.642197 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.642176    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-data\") pod \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\" (UID: \"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6\") "
Nov 03 10:48:23.642471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.642438    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-data" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:23.643268 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.643247    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:23.643363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.643271    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:23.643363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.643285    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:23.643363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.643298    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:23.643363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.643311    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:23.644622 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.644510    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-token" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:23.645039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.645003    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-podinfo" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:48:23.645612 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.645482    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-kube-api-access-7mxtf" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "kube-api-access-7mxtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:23.645956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.645793    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-envoy" pod "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" (UID: "83c991cb-9d1c-4377-b3e1-8e34e4eb25a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:23.744847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.744807    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:23.744847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.744839    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mxtf\" (UniqueName: \"kubernetes.io/projected/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-kube-api-access-7mxtf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:23.744847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.744849    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:23.744847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:23.744858    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.115376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.114800    5393 generic.go:356] "Generic (PLEG): container finished" podID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerID="09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c" exitCode=0
Nov 03 10:48:24.115376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.114878    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"
Nov 03 10:48:24.116609 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.116329    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" event={"ID":"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6","Type":"ContainerDied","Data":"09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c"}
Nov 03 10:48:24.116609 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.116391    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6" event={"ID":"83c991cb-9d1c-4377-b3e1-8e34e4eb25a6","Type":"ContainerDied","Data":"5b15191b3f2fc0e41a9ed30f9eda70b65b8ca354877fbd76248632b1a6ce90fd"}
Nov 03 10:48:24.116609 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.116421    5393 scope.go:122] "RemoveContainer" containerID="09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c"
Nov 03 10:48:24.122519 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.122497    5393 generic.go:356] "Generic (PLEG): container finished" podID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerID="a4e70440a5c7d1645c09bf9b4fb1243782d0ca22481873dc94ff42e6265c2d54" exitCode=0
Nov 03 10:48:24.122755 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.122740    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" event={"ID":"ac04ed6c-cedf-460c-b512-a9a2380a9793","Type":"ContainerDied","Data":"a4e70440a5c7d1645c09bf9b4fb1243782d0ca22481873dc94ff42e6265c2d54"}
Nov 03 10:48:24.133370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.133347    5393 scope.go:122] "RemoveContainer" containerID="1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0"
Nov 03 10:48:24.145502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.145424    5393 scope.go:122] "RemoveContainer" containerID="09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c"
Nov 03 10:48:24.145833 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:48:24.145808    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c\": container with ID starting with 09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c not found: ID does not exist" containerID="09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c"
Nov 03 10:48:24.145992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.145851    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c"} err="failed to get container status \"09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c\": rpc error: code = NotFound desc = could not find container \"09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c\": container with ID starting with 09d1b7bb59184be1725c3ba673a44e79936bd7065a7187496b90e4d3e571502c not found: ID does not exist"
Nov 03 10:48:24.145992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.145875    5393 scope.go:122] "RemoveContainer" containerID="1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0"
Nov 03 10:48:24.146203 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:48:24.146178    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0\": container with ID starting with 1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0 not found: ID does not exist" containerID="1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0"
Nov 03 10:48:24.146260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.146213    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0"} err="failed to get container status \"1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0\": rpc error: code = NotFound desc = could not find container \"1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0\": container with ID starting with 1cc29ef49f46525a2222e7a497b5ad39ce245f2643af8be87e60acccf83e0eb0 not found: ID does not exist"
Nov 03 10:48:24.146260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.146237    5393 scope.go:122] "RemoveContainer" containerID="156a5e1b670664db78fb3b2fb5a340bace8e8aa50b371ca142c17010f70633b6"
Nov 03 10:48:24.148982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.147219    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"]
Nov 03 10:48:24.155747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.155720    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7d8392-openshift-default-7cf6f4f76c-wxcj6"]
Nov 03 10:48:24.191036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.191009    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.283408    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-socket\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.283487    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-token\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-token\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.283555    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-data\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.283662    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-podinfo\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.283687    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-kube-api-access-6q2dj\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-kube-api-access-6q2dj\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.283778    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac04ed6c-cedf-460c-b512-a9a2380a9793-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac04ed6c-cedf-460c-b512-a9a2380a9793-istiod-ca-cert\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.284001    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-credential-socket\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.284083    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-envoy\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.285545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.284113    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-certs\") pod \"ac04ed6c-cedf-460c-b512-a9a2380a9793\" (UID: \"ac04ed6c-cedf-460c-b512-a9a2380a9793\") "
Nov 03 10:48:24.286533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.286505    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac04ed6c-cedf-460c-b512-a9a2380a9793-istiod-ca-cert" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:48:24.286787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.286767    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-socket" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:24.306816 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.306504    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-certs" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:24.361987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.317680    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-envoy" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:24.361987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.336087    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-data" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:24.361987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.337230    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-credential-socket" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:48:24.390171 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.390133    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" path="/var/lib/kubelet/pods/83c991cb-9d1c-4377-b3e1-8e34e4eb25a6/volumes"
Nov 03 10:48:24.390901 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.390880    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.391452 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.391435    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.391571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.391559    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.391665 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.391654    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.391760 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.391749    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.391841 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.391829    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac04ed6c-cedf-460c-b512-a9a2380a9793-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.397190 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.397161    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-token" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:24.400531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.400506    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-kube-api-access-6q2dj" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "kube-api-access-6q2dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:24.409944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.408828    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-podinfo" pod "ac04ed6c-cedf-460c-b512-a9a2380a9793" (UID: "ac04ed6c-cedf-460c-b512-a9a2380a9793"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:48:24.456256 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.456229    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:24.494444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.494413    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.494444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.494439    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac04ed6c-cedf-460c-b512-a9a2380a9793-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.494444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.494450    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q2dj\" (UniqueName: \"kubernetes.io/projected/ac04ed6c-cedf-460c-b512-a9a2380a9793-kube-api-access-6q2dj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.595693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.595596    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23750ab7-5579-47ce-9168-3e02effb1e3d-kube-api-access-2k4x6\" (UniqueName: \"kubernetes.io/projected/23750ab7-5579-47ce-9168-3e02effb1e3d-kube-api-access-2k4x6\") pod \"23750ab7-5579-47ce-9168-3e02effb1e3d\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") "
Nov 03 10:48:24.603490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.603458    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-openshift-ca\") pod \"23750ab7-5579-47ce-9168-3e02effb1e3d\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") "
Nov 03 10:48:24.603728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.603711    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-xks-ca\") pod \"23750ab7-5579-47ce-9168-3e02effb1e3d\" (UID: \"23750ab7-5579-47ce-9168-3e02effb1e3d\") "
Nov 03 10:48:24.606032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.605998    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-openshift-ca" pod "23750ab7-5579-47ce-9168-3e02effb1e3d" (UID: "23750ab7-5579-47ce-9168-3e02effb1e3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:48:24.608971 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.608942    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-xks-ca" pod "23750ab7-5579-47ce-9168-3e02effb1e3d" (UID: "23750ab7-5579-47ce-9168-3e02effb1e3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:48:24.609105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.609085    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23750ab7-5579-47ce-9168-3e02effb1e3d-kube-api-access-2k4x6" pod "23750ab7-5579-47ce-9168-3e02effb1e3d" (UID: "23750ab7-5579-47ce-9168-3e02effb1e3d"). InnerVolumeSpecName "kube-api-access-2k4x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:24.709293 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.709249    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k4x6\" (UniqueName: \"kubernetes.io/projected/23750ab7-5579-47ce-9168-3e02effb1e3d-kube-api-access-2k4x6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.709836 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.709813    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:24.709988 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:24.709973    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23750ab7-5579-47ce-9168-3e02effb1e3d-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:25.130511 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:25.130482    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"
Nov 03 10:48:25.130702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:25.130508    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n" event={"ID":"23750ab7-5579-47ce-9168-3e02effb1e3d","Type":"ContainerDied","Data":"721ebd3aa847c55e1ca07b8f3d6bd86ad16f0297eb6b8fa469c52aba35a47953"}
Nov 03 10:48:25.130702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:25.130604    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721ebd3aa847c55e1ca07b8f3d6bd86ad16f0297eb6b8fa469c52aba35a47953"
Nov 03 10:48:25.133153 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:25.133131    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"
Nov 03 10:48:25.133279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:25.133161    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92" event={"ID":"ac04ed6c-cedf-460c-b512-a9a2380a9793","Type":"ContainerDied","Data":"d881197fb4160a1b4ea909caea4d92f2d8f6fa9d655b0f3cf6d992d54a3fc224"}
Nov 03 10:48:25.133279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:25.133213    5393 scope.go:122] "RemoveContainer" containerID="a4e70440a5c7d1645c09bf9b4fb1243782d0ca22481873dc94ff42e6265c2d54"
Nov 03 10:48:25.222956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:25.221975    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"]
Nov 03 10:48:25.239336 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:25.239267    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-273012-openshift-default-696fb5cfbf-nxk92"]
Nov 03 10:48:26.241597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:26.241559    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" path="/var/lib/kubelet/pods/ac04ed6c-cedf-460c-b512-a9a2380a9793/volumes"
Nov 03 10:48:27.751517 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.751489    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"]
Nov 03 10:48:27.752121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752097    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy"
Nov 03 10:48:27.752121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752115    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752132    5393 cpu_manager.go:414] "containerMap: removing container" podUID="23750ab7-5579-47ce-9168-3e02effb1e3d" containerName="revoke-keys"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752137    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23750ab7-5579-47ce-9168-3e02effb1e3d" containerName="revoke-keys"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752145    5393 cpu_manager.go:414] "containerMap: removing container" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752149    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752171    5393 cpu_manager.go:414] "containerMap: removing container" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752175    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752271    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752282    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy"
Nov 03 10:48:27.752287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752289    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy"
Nov 03 10:48:27.752537 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.752295    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="23750ab7-5579-47ce-9168-3e02effb1e3d" containerName="revoke-keys"
Nov 03 10:48:27.757435 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.757412    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.761030 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.761002    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:48:27.762338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.762313    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:48:27.763977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.763952    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"]
Nov 03 10:48:27.793949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.792955    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdc8c\" (UniqueName: \"kubernetes.io/projected/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-kube-api-access-zdc8c\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.793949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.793044    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.793949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.793320    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.894936 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.894891    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.895113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.895009    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdc8c\" (UniqueName: \"kubernetes.io/projected/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-kube-api-access-zdc8c\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.895113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.895068    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.895694 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.895674    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.897297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.897274    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:27.903247 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:27.903223    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdc8c\" (UniqueName: \"kubernetes.io/projected/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-kube-api-access-zdc8c\") pod \"maas-api-revoke-keys-e2e-api-b-642fe5-vg927\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:28.079304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:28.079220    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:28.365379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:28.365114    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"]
Nov 03 10:48:29.166565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.166523    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927" event={"ID":"9b2fcc3d-64c7-4bb0-90d7-603b162e964c","Type":"ContainerStarted","Data":"6f8d8b6ae003072564ea63663f250ef32ea076649163257889d243e84025b2f0"}
Nov 03 10:48:29.167055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.166591    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927" event={"ID":"9b2fcc3d-64c7-4bb0-90d7-603b162e964c","Type":"ContainerStarted","Data":"ba16ce55946afbd8bae241196b990dab144cd8e24810f4b929a3383246768369"}
Nov 03 10:48:29.183298 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.183242    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927" podStartSLOduration=2.18322616 podStartE2EDuration="2.18322616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:29.181452978 +0000 UTC m=+1153.086307419" watchObservedRunningTime="2026-11-03 10:48:29.18322616 +0000 UTC m=+1153.088080585"
Nov 03 10:48:29.861070 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.861042    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"]
Nov 03 10:48:29.862015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.861992    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy"
Nov 03 10:48:29.862077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.862016    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac04ed6c-cedf-460c-b512-a9a2380a9793" containerName="istio-proxy"
Nov 03 10:48:29.862196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.862181    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c991cb-9d1c-4377-b3e1-8e34e4eb25a6" containerName="istio-proxy"
Nov 03 10:48:29.866277 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.866255    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:29.885247 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.885209    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-90bad4-serving-cert\""
Nov 03 10:48:29.892319 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.892285    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"]
Nov 03 10:48:29.928447 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.928408    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-openshift-service-ca\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:29.928626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.928472    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-metrics-certs\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:29.928626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.928541    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-maas-api-tls\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:29.928775 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:29.928630    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2lr8\" (UniqueName: \"kubernetes.io/projected/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-kube-api-access-l2lr8\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.038997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.038828    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2lr8\" (UniqueName: \"kubernetes.io/projected/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-kube-api-access-l2lr8\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.039209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.039143    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-openshift-service-ca\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.039276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.039227    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-metrics-certs\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.039332 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.039303    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-maas-api-tls\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.041709 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.040223    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-openshift-service-ca\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.047960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.047859    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-maas-api-tls\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.067143 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.067102    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2lr8\" (UniqueName: \"kubernetes.io/projected/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-kube-api-access-l2lr8\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.100975 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.090420    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-metrics-certs\") pod \"maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.174643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.174552    5393 generic.go:356] "Generic (PLEG): container finished" podID="9b2fcc3d-64c7-4bb0-90d7-603b162e964c" containerID="6f8d8b6ae003072564ea63663f250ef32ea076649163257889d243e84025b2f0" exitCode=0
Nov 03 10:48:30.174643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.174627    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927" event={"ID":"9b2fcc3d-64c7-4bb0-90d7-603b162e964c","Type":"ContainerDied","Data":"6f8d8b6ae003072564ea63663f250ef32ea076649163257889d243e84025b2f0"}
Nov 03 10:48:30.200884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.200855    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:30.483942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:30.483836    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"]
Nov 03 10:48:30.553300 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:48:30.547227    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100f9b59_5a15_4084_94bf_d9fe0cf1c6ab.slice/crio-5d50075f875c069dc2ea53383ff95fad3fa9fee595ae63f08674635a208c0100 WatchSource:0}: Error finding container 5d50075f875c069dc2ea53383ff95fad3fa9fee595ae63f08674635a208c0100: Status 404 returned error can't find the container with id 5d50075f875c069dc2ea53383ff95fad3fa9fee595ae63f08674635a208c0100
Nov 03 10:48:31.182887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.182831    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c" event={"ID":"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab","Type":"ContainerStarted","Data":"1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286"}
Nov 03 10:48:31.182887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.182893    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c" event={"ID":"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab","Type":"ContainerStarted","Data":"5d50075f875c069dc2ea53383ff95fad3fa9fee595ae63f08674635a208c0100"}
Nov 03 10:48:31.183406 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.183176    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:31.201062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.201013    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c" podStartSLOduration=1.726890707 podStartE2EDuration="2.200996906s" totalImagesPullingTime="474.106199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:31.198541654 +0000 UTC m=+1155.103396095" watchObservedRunningTime="2026-11-03 10:48:31.200996906 +0000 UTC m=+1155.105851347"
Nov 03 10:48:31.435899 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.435875    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:31.510701 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.510664    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-openshift-ca\") pod \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") "
Nov 03 10:48:31.510857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.510721    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-xks-ca\") pod \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") "
Nov 03 10:48:31.510857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.510851    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-kube-api-access-zdc8c\" (UniqueName: \"kubernetes.io/projected/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-kube-api-access-zdc8c\") pod \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\" (UID: \"9b2fcc3d-64c7-4bb0-90d7-603b162e964c\") "
Nov 03 10:48:31.511079 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.511045    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-openshift-ca" pod "9b2fcc3d-64c7-4bb0-90d7-603b162e964c" (UID: "9b2fcc3d-64c7-4bb0-90d7-603b162e964c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:48:31.511376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.511357    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:31.513012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.512987    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-xks-ca" pod "9b2fcc3d-64c7-4bb0-90d7-603b162e964c" (UID: "9b2fcc3d-64c7-4bb0-90d7-603b162e964c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:48:31.513233 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.513211    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-kube-api-access-zdc8c" pod "9b2fcc3d-64c7-4bb0-90d7-603b162e964c" (UID: "9b2fcc3d-64c7-4bb0-90d7-603b162e964c"). InnerVolumeSpecName "kube-api-access-zdc8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:31.612667 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.612632    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:31.612667 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:31.612663    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdc8c\" (UniqueName: \"kubernetes.io/projected/9b2fcc3d-64c7-4bb0-90d7-603b162e964c-kube-api-access-zdc8c\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:32.305943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:32.299716    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"
Nov 03 10:48:32.333781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:32.333733    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927" event={"ID":"9b2fcc3d-64c7-4bb0-90d7-603b162e964c","Type":"ContainerDied","Data":"ba16ce55946afbd8bae241196b990dab144cd8e24810f4b929a3383246768369"}
Nov 03 10:48:32.335787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:32.335765    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba16ce55946afbd8bae241196b990dab144cd8e24810f4b929a3383246768369"
Nov 03 10:48:33.793339 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.793296    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"]
Nov 03 10:48:33.794792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.794754    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9b2fcc3d-64c7-4bb0-90d7-603b162e964c" containerName="revoke-keys"
Nov 03 10:48:33.794982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.794966    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2fcc3d-64c7-4bb0-90d7-603b162e964c" containerName="revoke-keys"
Nov 03 10:48:33.795299 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.795286    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2fcc3d-64c7-4bb0-90d7-603b162e964c" containerName="revoke-keys"
Nov 03 10:48:33.799370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.799348    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.806495 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.806470    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a51d55-openshift-default-dockercfg-rpb7t\""
Nov 03 10:48:33.809624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.809594    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"]
Nov 03 10:48:33.874488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.874452    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbq6n\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-kube-api-access-jbq6n\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.874674 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.874646    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-data\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.874735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.874690    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-credential-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.874835 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.874816    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istiod-ca-cert\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.874900 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.874860    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.875023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.875008    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-podinfo\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.875088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.875062    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-token\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.875158 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.875142    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-certs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.875215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.875188    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-envoy\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.988349 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.988311    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbq6n\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-kube-api-access-jbq6n\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.989084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.989058    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-data\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.989700 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.989679    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-credential-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.990168 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.990150    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istiod-ca-cert\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.996013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.995976    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.996569 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.996549    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-podinfo\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:33.997199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.997180    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-token\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.019465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.019437    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-certs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.020028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.020004    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-envoy\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.020318 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.989603    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-data\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.020421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.990869    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istiod-ca-cert\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.020513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.019944    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-certs\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.020598 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.990072    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-credential-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.020686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:33.996419    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-socket\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.037331 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.037265    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-envoy\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.046982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.040736    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbq6n\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-kube-api-access-jbq6n\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.047739 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.047713    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-podinfo\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.062945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.048338    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-token\") pod \"e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") " pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.131095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.131063    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:34.541296 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.541267    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"]
Nov 03 10:48:34.542288 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:48:34.542261    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeaca4fd_b4cd_4e53_8854_5cd4c1bd8356.slice/crio-34312f566b227bd1176717f8371c642fc8642cb22ec4498a89ca02e38f0f0117 WatchSource:0}: Error finding container 34312f566b227bd1176717f8371c642fc8642cb22ec4498a89ca02e38f0f0117: Status 404 returned error can't find the container with id 34312f566b227bd1176717f8371c642fc8642cb22ec4498a89ca02e38f0f0117
Nov 03 10:48:34.545636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.545599    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:34.545703 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.545687    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:34.545747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:34.545730    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:35.441478 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:35.441442    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" event={"ID":"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356","Type":"ContainerStarted","Data":"027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671"}
Nov 03 10:48:35.441996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:35.441496    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" event={"ID":"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356","Type":"ContainerStarted","Data":"34312f566b227bd1176717f8371c642fc8642cb22ec4498a89ca02e38f0f0117"}
Nov 03 10:48:35.466238 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:35.466185    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podStartSLOduration=2.466169705 podStartE2EDuration="2.466169705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:35.461045176 +0000 UTC m=+1159.365899618" watchObservedRunningTime="2026-11-03 10:48:35.466169705 +0000 UTC m=+1159.371024145"
Nov 03 10:48:36.132449 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:36.132413    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:36.137443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:36.137419    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:36.449199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:36.449119    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:36.450219 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:36.450200    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:48:37.306413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:37.306334    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:48:41.065948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:41.065017    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-64cfdb"]
Nov 03 10:48:41.076945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:41.074184    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:41.080943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:41.078702    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5zm6k\""
Nov 03 10:48:41.106354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:41.106317    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-64cfdb"]
Nov 03 10:48:41.184008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:41.183962    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpc26\" (UniqueName: \"kubernetes.io/projected/eb09904b-06b5-4276-a621-9a4d5d2fcf7b-kube-api-access-wpc26\") pod \"test-curl-7603-64cfdb\" (UID: \"eb09904b-06b5-4276-a621-9a4d5d2fcf7b\") " pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:41.313619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:41.313580    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpc26\" (UniqueName: \"kubernetes.io/projected/eb09904b-06b5-4276-a621-9a4d5d2fcf7b-kube-api-access-wpc26\") pod \"test-curl-7603-64cfdb\" (UID: \"eb09904b-06b5-4276-a621-9a4d5d2fcf7b\") " pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:41.469361 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:41.469326    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpc26\" (UniqueName: \"kubernetes.io/projected/eb09904b-06b5-4276-a621-9a4d5d2fcf7b-kube-api-access-wpc26\") pod \"test-curl-7603-64cfdb\" (UID: \"eb09904b-06b5-4276-a621-9a4d5d2fcf7b\") " pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:41.613378 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:41.613337    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:42.102408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:42.102370    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-64cfdb"]
Nov 03 10:48:42.104876 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:48:42.104849    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb09904b_06b5_4276_a621_9a4d5d2fcf7b.slice/crio-c9cbc11e62d7356efc1cd56a49148a32be30546294513024a97cc0b9ebde2591 WatchSource:0}: Error finding container c9cbc11e62d7356efc1cd56a49148a32be30546294513024a97cc0b9ebde2591: Status 404 returned error can't find the container with id c9cbc11e62d7356efc1cd56a49148a32be30546294513024a97cc0b9ebde2591
Nov 03 10:48:42.637960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:42.637903    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-64cfdb" event={"ID":"eb09904b-06b5-4276-a621-9a4d5d2fcf7b","Type":"ContainerStarted","Data":"bcc0fd3fedf8457e94aa4dd718bb94baddaa374fc1f5ab6348a16db3d70ab6d1"}
Nov 03 10:48:42.637960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:42.637960    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-64cfdb" event={"ID":"eb09904b-06b5-4276-a621-9a4d5d2fcf7b","Type":"ContainerStarted","Data":"c9cbc11e62d7356efc1cd56a49148a32be30546294513024a97cc0b9ebde2591"}
Nov 03 10:48:42.652747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:42.652691    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7603-64cfdb" podStartSLOduration=1.652674764 podStartE2EDuration="1.652674764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:42.65216098 +0000 UTC m=+1166.557015430" watchObservedRunningTime="2026-11-03 10:48:42.652674764 +0000 UTC m=+1166.557529207"
Nov 03 10:48:43.252055 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:48:43.252011    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:48:43.432353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:43.432315    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-64cfdb"]
Nov 03 10:48:43.436873 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:43.436839    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-64cfdb"]
Nov 03 10:48:44.651529 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:44.651480    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7603-64cfdb" podUID="eb09904b-06b5-4276-a621-9a4d5d2fcf7b" containerName="test-curl-7603-64cfdb" containerID="cri-o://bcc0fd3fedf8457e94aa4dd718bb94baddaa374fc1f5ab6348a16db3d70ab6d1" gracePeriod=2
Nov 03 10:48:45.492107 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.492066    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"]
Nov 03 10:48:45.493268 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.493243    5393 cpu_manager.go:414] "containerMap: removing container" podUID="eb09904b-06b5-4276-a621-9a4d5d2fcf7b" containerName="test-curl-7603-64cfdb"
Nov 03 10:48:45.501277 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.501248    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb09904b-06b5-4276-a621-9a4d5d2fcf7b" containerName="test-curl-7603-64cfdb"
Nov 03 10:48:45.501499 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.501482    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb09904b-06b5-4276-a621-9a4d5d2fcf7b" containerName="test-curl-7603-64cfdb"
Nov 03 10:48:45.507404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.507375    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.511162 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.507273    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"]
Nov 03 10:48:45.512485 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.512431    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-38c9b0-openshift-default-dockercfg-2zxgv\""
Nov 03 10:48:45.539719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.539686    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-certs\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.539719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.539722    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-envoy\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.539987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.539826    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-credential-socket\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.539987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.539858    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wnj\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-kube-api-access-m9wnj\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.540099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.539987    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-podinfo\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.540099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.540027    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istiod-ca-cert\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.540195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.540096    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-socket\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.540244 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.540220    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-token\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.540291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.540271    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-data\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.645859    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-certs\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.645932    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-envoy\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.646002    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-credential-socket\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.646043    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wnj\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-kube-api-access-m9wnj\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.646104    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-podinfo\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.646156    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istiod-ca-cert\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.646195    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-socket\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.646315    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-token\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.646388    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-data\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.646897    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-data\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.657944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.647166    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-certs\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.685945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.662270    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-socket\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.685945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.662876    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istiod-ca-cert\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.685945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.666412    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-credential-socket\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.685945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.682679    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-podinfo\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.685945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.683212    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-envoy\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.763989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.760186    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wnj\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-kube-api-access-m9wnj\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.763989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.760580    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-token\") pod \"e2e-api-b-38c9b0-openshift-default-778969b477-d74m2\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") " pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:45.829956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:45.828214    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:46.411690 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:48:46.411662    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04b7efd_1fdb_4ac2_b9bf_13765816ddd0.slice/crio-7e675b8d8e540e904d33c846912ec98863cb9615ef9be789c071a60201e9564e WatchSource:0}: Error finding container 7e675b8d8e540e904d33c846912ec98863cb9615ef9be789c071a60201e9564e: Status 404 returned error can't find the container with id 7e675b8d8e540e904d33c846912ec98863cb9615ef9be789c071a60201e9564e
Nov 03 10:48:46.413977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.413948    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"]
Nov 03 10:48:46.414393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.414363    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:46.414443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.414432    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:46.414477 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.414465    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:48:46.756862 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.756366    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" event={"ID":"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0","Type":"ContainerStarted","Data":"09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff"}
Nov 03 10:48:46.756862 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.756436    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" event={"ID":"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0","Type":"ContainerStarted","Data":"7e675b8d8e540e904d33c846912ec98863cb9615ef9be789c071a60201e9564e"}
Nov 03 10:48:46.759602 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.759569    5393 generic.go:356] "Generic (PLEG): container finished" podID="eb09904b-06b5-4276-a621-9a4d5d2fcf7b" containerID="bcc0fd3fedf8457e94aa4dd718bb94baddaa374fc1f5ab6348a16db3d70ab6d1" exitCode=137
Nov 03 10:48:46.792721 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.786572    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podStartSLOduration=1.786550843 podStartE2EDuration="1.786550843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:48:46.783667759 +0000 UTC m=+1170.688522201" watchObservedRunningTime="2026-11-03 10:48:46.786550843 +0000 UTC m=+1170.691405285"
Nov 03 10:48:46.828881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:46.828853    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:47.036998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:47.029338    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:47.132310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:47.131522    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb09904b-06b5-4276-a621-9a4d5d2fcf7b-kube-api-access-wpc26\" (UniqueName: \"kubernetes.io/projected/eb09904b-06b5-4276-a621-9a4d5d2fcf7b-kube-api-access-wpc26\") pod \"eb09904b-06b5-4276-a621-9a4d5d2fcf7b\" (UID: \"eb09904b-06b5-4276-a621-9a4d5d2fcf7b\") "
Nov 03 10:48:47.135645 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:47.135616    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb09904b-06b5-4276-a621-9a4d5d2fcf7b-kube-api-access-wpc26" pod "eb09904b-06b5-4276-a621-9a4d5d2fcf7b" (UID: "eb09904b-06b5-4276-a621-9a4d5d2fcf7b"). InnerVolumeSpecName "kube-api-access-wpc26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:48:47.234186 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:47.234151    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpc26\" (UniqueName: \"kubernetes.io/projected/eb09904b-06b5-4276-a621-9a4d5d2fcf7b-kube-api-access-wpc26\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:48:47.771730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:47.771699    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:47.772483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:47.771718    5393 scope.go:122] "RemoveContainer" containerID="bcc0fd3fedf8457e94aa4dd718bb94baddaa374fc1f5ab6348a16db3d70ab6d1"
Nov 03 10:48:47.787554 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:48:47.787514    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7603-64cfdb\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="eb09904b-06b5-4276-a621-9a4d5d2fcf7b" pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:47.835284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:47.835254    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:47.846942 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:48:47.842339    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7603-64cfdb\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="eb09904b-06b5-4276-a621-9a4d5d2fcf7b" pod="opendatahub/test-curl-7603-64cfdb"
Nov 03 10:48:48.230404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:48.230369    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb09904b-06b5-4276-a621-9a4d5d2fcf7b" path="/var/lib/kubelet/pods/eb09904b-06b5-4276-a621-9a4d5d2fcf7b/volumes"
Nov 03 10:48:48.812018 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:48.810580    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:48.822233 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:48.822027    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:48:53.971026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:53.970989    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"]
Nov 03 10:48:53.987823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:53.987782    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-642fe5-vg927"]
Nov 03 10:48:54.253656 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:48:54.253561    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2fcc3d-64c7-4bb0-90d7-603b162e964c" path="/var/lib/kubelet/pods/9b2fcc3d-64c7-4bb0-90d7-603b162e964c/volumes"
Nov 03 10:49:04.163260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.163226    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"]
Nov 03 10:49:04.182821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.182790    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.198981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.198949    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4d9549-openshift-default-dockercfg-zgsp6\""
Nov 03 10:49:04.288683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.288648    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"]
Nov 03 10:49:04.300203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300173    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-certs\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.300374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300241    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-credential-socket\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.300374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300319    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-socket\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.300374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300351    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-token\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.300535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300442    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-podinfo\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.300535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300487    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhhgt\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-kube-api-access-jhhgt\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.300645 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300550    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-envoy\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.300645 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300569    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-data\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.300747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.300682    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8395d93-9b56-4476-a706-3e00b3847ef6-istiod-ca-cert\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.401634 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.401594    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-envoy\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.401828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.401643    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-data\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.401828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.401727    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8395d93-9b56-4476-a706-3e00b3847ef6-istiod-ca-cert\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.401956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.401877    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-certs\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.401962    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-credential-socket\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402024    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-socket\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402051    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-token\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402122    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-podinfo\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402158    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhhgt\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-kube-api-access-jhhgt\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402193    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-data\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402392 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402371    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-certs\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402505    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-credential-socket\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402824    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8395d93-9b56-4476-a706-3e00b3847ef6-istiod-ca-cert\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.402995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.402976    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-socket\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.405258 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.405236    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-podinfo\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.407709 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.407685    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-envoy\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.415364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.415302    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-token\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.415803 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.415783    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhhgt\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-kube-api-access-jhhgt\") pod \"e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") " pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.572946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.566200    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:04.761070 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:04.760076    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"]
Nov 03 10:49:05.095714 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:05.095447    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"]
Nov 03 10:49:05.099272 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:49:05.099214    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8395d93_9b56_4476_a706_3e00b3847ef6.slice/crio-7e1db6961d04cd7032a424441d1e88e91f155e852e7823c06485ac2be6cd677c WatchSource:0}: Error finding container 7e1db6961d04cd7032a424441d1e88e91f155e852e7823c06485ac2be6cd677c: Status 404 returned error can't find the container with id 7e1db6961d04cd7032a424441d1e88e91f155e852e7823c06485ac2be6cd677c
Nov 03 10:49:05.103224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:05.103187    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:05.103377 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:05.103276    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:05.103377 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:05.103317    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:05.177978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:05.177236    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" event={"ID":"d8395d93-9b56-4476-a706-3e00b3847ef6","Type":"ContainerStarted","Data":"7e1db6961d04cd7032a424441d1e88e91f155e852e7823c06485ac2be6cd677c"}
Nov 03 10:49:06.190077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:06.190033    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" event={"ID":"d8395d93-9b56-4476-a706-3e00b3847ef6","Type":"ContainerStarted","Data":"210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da"}
Nov 03 10:49:06.567436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:06.567336    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:06.574078 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:06.574039    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:06.598155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:06.598089    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" podStartSLOduration=2.598067749 podStartE2EDuration="2.598067749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:06.21325884 +0000 UTC m=+1190.118113283" watchObservedRunningTime="2026-11-03 10:49:06.598067749 +0000 UTC m=+1190.502922193"
Nov 03 10:49:07.198602 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:07.198566    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:07.199749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:07.199728    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:49:09.149130 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:09.149048    5393 patch_prober.go:29] interesting pod/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:49:09.149130 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:09.149108    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:49:10.298580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:10.291832    5393 generic.go:356] "Generic (PLEG): container finished" podID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerID="ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e" exitCode=0
Nov 03 10:49:10.298580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:10.292035    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" event={"ID":"28d4380b-8e6e-4505-a07e-4bf980ec7296","Type":"ContainerDied","Data":"ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e"}
Nov 03 10:49:10.298580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:10.292581    5393 scope.go:122] "RemoveContainer" containerID="ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e"
Nov 03 10:49:10.298580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:10.295285    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:10.298580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:10.295408    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:10.298580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:10.295453    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:11.301682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:11.301647    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" event={"ID":"28d4380b-8e6e-4505-a07e-4bf980ec7296","Type":"ContainerStarted","Data":"c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0"}
Nov 03 10:49:11.642946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:11.642290    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:49:11.642946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:11.642424    5393 patch_prober.go:29] interesting pod/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 10:49:11.642946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:11.642466    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 10:49:12.631598 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:12.631562    5393 patch_prober.go:29] interesting pod/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 10:49:12.632035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:12.631625    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 10:49:13.216382 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:13.216343    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:49:13.637349 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:13.637316    5393 patch_prober.go:29] interesting pod/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 10:49:13.638022 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:13.637984    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 10:49:14.631015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:14.630983    5393 patch_prober.go:29] interesting pod/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 10:49:14.631199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:14.631054    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 10:49:15.632945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:15.631380    5393 patch_prober.go:29] interesting pod/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 10:49:15.632945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:15.631437    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 10:49:15.964026 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:15.960038    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:49:16.635138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:16.635104    5393 patch_prober.go:29] interesting pod/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 10:49:16.635734 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:16.635700    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 10:49:16.679630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:16.671830    5393 scope.go:122] "RemoveContainer" containerID="3dcbc3d54871b141678b87937b9dd0d83eb9de7f04c9fb9438a58a2ca8c6339d"
Nov 03 10:49:17.004722 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:17.003615    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:49:17.054905 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:17.054829    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:49:17.332402 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:17.331692    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:49:17.629940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:17.629077    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:49:17.658943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:17.658814    5393 patch_prober.go:29] interesting pod/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 10:49:17.658943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:17.658879    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 10:49:17.777942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:17.771763    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:49:17.797585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:17.796125    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:49:18.749931 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:18.749871    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:49:18.750435 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:18.750328    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:49:18.751422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:18.751400    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:49:19.845142 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:19.845020    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x" podUID="f231c2ce-f4b1-48c3-829e-a10e26e22960" containerName="main" containerID="cri-o://f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4" gracePeriod=45
Nov 03 10:49:20.347013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.340022    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x" podUID="f231c2ce-f4b1-48c3-829e-a10e26e22960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": dial tcp 10.134.0.82:8000: connect: connection refused"
Nov 03 10:49:20.672203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.672173    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:49:20.775425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.775034    5393 generic.go:356] "Generic (PLEG): container finished" podID="f231c2ce-f4b1-48c3-829e-a10e26e22960" containerID="f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4" exitCode=0
Nov 03 10:49:20.775425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.775086    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"
Nov 03 10:49:20.775425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.775259    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x" event={"ID":"f231c2ce-f4b1-48c3-829e-a10e26e22960","Type":"ContainerDied","Data":"f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4"}
Nov 03 10:49:20.775425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.775291    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x" event={"ID":"f231c2ce-f4b1-48c3-829e-a10e26e22960","Type":"ContainerDied","Data":"a58f8bdbb19d09132af0cf52396a667d832d054fcece708f8aa88d505e5a4063"}
Nov 03 10:49:20.775425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.775311    5393 scope.go:122] "RemoveContainer" containerID="f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4"
Nov 03 10:49:20.798942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.781347    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-dshm\") pod \"f231c2ce-f4b1-48c3-829e-a10e26e22960\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") "
Nov 03 10:49:20.798942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.781433    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-model-cache\") pod \"f231c2ce-f4b1-48c3-829e-a10e26e22960\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") "
Nov 03 10:49:20.798942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.781507    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f231c2ce-f4b1-48c3-829e-a10e26e22960-kube-api-access-tf5vj\" (UniqueName: \"kubernetes.io/projected/f231c2ce-f4b1-48c3-829e-a10e26e22960-kube-api-access-tf5vj\") pod \"f231c2ce-f4b1-48c3-829e-a10e26e22960\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") "
Nov 03 10:49:20.798942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.781597    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-home\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-home\") pod \"f231c2ce-f4b1-48c3-829e-a10e26e22960\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") "
Nov 03 10:49:20.798942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.781726    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f231c2ce-f4b1-48c3-829e-a10e26e22960-tls-certs\" (UniqueName: \"kubernetes.io/secret/f231c2ce-f4b1-48c3-829e-a10e26e22960-tls-certs\") pod \"f231c2ce-f4b1-48c3-829e-a10e26e22960\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") "
Nov 03 10:49:20.798942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.781783    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-tmp-dir\") pod \"f231c2ce-f4b1-48c3-829e-a10e26e22960\" (UID: \"f231c2ce-f4b1-48c3-829e-a10e26e22960\") "
Nov 03 10:49:20.825275 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.825216    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-model-cache" pod "f231c2ce-f4b1-48c3-829e-a10e26e22960" (UID: "f231c2ce-f4b1-48c3-829e-a10e26e22960"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:20.825601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.825559    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-home" pod "f231c2ce-f4b1-48c3-829e-a10e26e22960" (UID: "f231c2ce-f4b1-48c3-829e-a10e26e22960"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:20.855071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.855035    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-tmp-dir" pod "f231c2ce-f4b1-48c3-829e-a10e26e22960" (UID: "f231c2ce-f4b1-48c3-829e-a10e26e22960"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:20.858082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.858018    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f231c2ce-f4b1-48c3-829e-a10e26e22960-tls-certs" pod "f231c2ce-f4b1-48c3-829e-a10e26e22960" (UID: "f231c2ce-f4b1-48c3-829e-a10e26e22960"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:49:20.858082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.858040    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f231c2ce-f4b1-48c3-829e-a10e26e22960-kube-api-access-tf5vj" pod "f231c2ce-f4b1-48c3-829e-a10e26e22960" (UID: "f231c2ce-f4b1-48c3-829e-a10e26e22960"). InnerVolumeSpecName "kube-api-access-tf5vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:20.859104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.859080    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-dshm" pod "f231c2ce-f4b1-48c3-829e-a10e26e22960" (UID: "f231c2ce-f4b1-48c3-829e-a10e26e22960"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:20.870781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.870761    5393 scope.go:122] "RemoveContainer" containerID="f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4"
Nov 03 10:49:20.871227 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:20.871195    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4\": container with ID starting with f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4 not found: ID does not exist" containerID="f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4"
Nov 03 10:49:20.871337 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.871238    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4"} err="failed to get container status \"f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4\": rpc error: code = NotFound desc = could not find container \"f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4\": container with ID starting with f01abdf00532614af88ce039a2f569a51d30f9ea4467edadfc1f27ad6fe801d4 not found: ID does not exist"
Nov 03 10:49:20.885891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.885857    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf5vj\" (UniqueName: \"kubernetes.io/projected/f231c2ce-f4b1-48c3-829e-a10e26e22960-kube-api-access-tf5vj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:20.887061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.887040    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:20.887222 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.887209    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f231c2ce-f4b1-48c3-829e-a10e26e22960-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:20.887319 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.887309    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:20.887411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.887400    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:20.887496 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:20.887485    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f231c2ce-f4b1-48c3-829e-a10e26e22960-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:21.048646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:21.048593    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"]
Nov 03 10:49:21.051502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:21.051478    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a91942-lgsl2"]
Nov 03 10:49:21.230948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:21.219725    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"]
Nov 03 10:49:21.258455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:21.258409    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ab592834/e2e-lifecycle-model-ab592834-kserve-74d56c4549-8bg6x"]
Nov 03 10:49:22.265214 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:22.265181    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1becc992-f950-4c6c-b71f-7f06e513f067" path="/var/lib/kubelet/pods/1becc992-f950-4c6c-b71f-7f06e513f067/volumes"
Nov 03 10:49:22.266461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:22.266434    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f231c2ce-f4b1-48c3-829e-a10e26e22960" path="/var/lib/kubelet/pods/f231c2ce-f4b1-48c3-829e-a10e26e22960/volumes"
Nov 03 10:49:25.278478 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.278440    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"]
Nov 03 10:49:25.279960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.279927    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f231c2ce-f4b1-48c3-829e-a10e26e22960" containerName="main"
Nov 03 10:49:25.280114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.280100    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f231c2ce-f4b1-48c3-829e-a10e26e22960" containerName="main"
Nov 03 10:49:25.280329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.280312    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f231c2ce-f4b1-48c3-829e-a10e26e22960" containerName="main"
Nov 03 10:49:25.284265 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.284241    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.292961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.292399    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-aa420c-openshift-default-dockercfg-j9vjf\""
Nov 03 10:49:25.294113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.294079    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"]
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439063    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9jm5\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-kube-api-access-n9jm5\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439137    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-certs\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439205    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-envoy\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439232    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-data\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439297    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-podinfo\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439324    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-token\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439495    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istiod-ca-cert\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439578    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-credential-socket\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.469982 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.439683    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-socket\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.545282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.545187    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-podinfo\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.546038    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-token\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.546201    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istiod-ca-cert\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.546276    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-credential-socket\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.546369    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-socket\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.546475    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9jm5\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-kube-api-access-n9jm5\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.546513    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-certs\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.546558    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-envoy\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.546578    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-data\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.547032    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-data\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.548445    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-credential-socket\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.548664    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-socket\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.556444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.548878    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-certs\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.572069 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.572035    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istiod-ca-cert\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.576311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.576262    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-envoy\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.602001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.581856    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-token\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.602001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.587096    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9jm5\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-kube-api-access-n9jm5\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.718756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.718683    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-podinfo\") pod \"e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") " pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:25.910311 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:25.910272    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:26.395023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:26.394974    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"]
Nov 03 10:49:26.415794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:26.415749    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:26.415898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:26.415840    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:26.415898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:26.415884    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:26.897946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:26.897097    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" event={"ID":"f78765ce-ebc8-4d04-8ae6-721dddd8ba72","Type":"ContainerStarted","Data":"8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1"}
Nov 03 10:49:26.897946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:26.897173    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" event={"ID":"f78765ce-ebc8-4d04-8ae6-721dddd8ba72","Type":"ContainerStarted","Data":"8f21b40529913ed14a9e5e59d8860ad5c70ceac3df1d9bc4f3eee515bede8a6a"}
Nov 03 10:49:26.918229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:26.918128    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:26.934187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:26.934130    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podStartSLOduration=1.934111484 podStartE2EDuration="1.934111484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:26.932785534 +0000 UTC m=+1210.837639987" watchObservedRunningTime="2026-11-03 10:49:26.934111484 +0000 UTC m=+1210.838965926"
Nov 03 10:49:27.917751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:27.917716    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:28.915020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.914987    5393 generic.go:356] "Generic (PLEG): container finished" podID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerID="027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671" exitCode=0
Nov 03 10:49:28.915228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.915055    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" event={"ID":"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356","Type":"ContainerDied","Data":"027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671"}
Nov 03 10:49:28.915520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.915501    5393 scope.go:122] "RemoveContainer" containerID="027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671"
Nov 03 10:49:28.916976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.916944    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:28.917098 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.917003    5393 generic.go:356] "Generic (PLEG): container finished" podID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerID="09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff" exitCode=0
Nov 03 10:49:28.917098 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.917027    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:28.917098 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.917035    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" event={"ID":"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0","Type":"ContainerDied","Data":"09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff"}
Nov 03 10:49:28.917098 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.917076    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:28.917317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.917298    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:28.917614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.917536    5393 scope.go:122] "RemoveContainer" containerID="09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff"
Nov 03 10:49:28.918983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.918960    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:49:28.919354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.918981    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:28.919354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.919054    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:28.919354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:28.919093    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:29.939489 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:29.937300    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" event={"ID":"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356","Type":"ContainerStarted","Data":"e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1"}
Nov 03 10:49:29.994943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:29.990073    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" event={"ID":"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0","Type":"ContainerStarted","Data":"d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a"}
Nov 03 10:49:30.132085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.132048    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:49:30.132265 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.132245    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:49:30.132315 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.132289    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 10:49:30.696346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.696309    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"]
Nov 03 10:49:30.699587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.699560    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:30.710257 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.709169    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa420c\""
Nov 03 10:49:30.711519 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.711498    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"]
Nov 03 10:49:30.865939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.856810    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:49:30.890840 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.883128    5393 patch_prober.go:29] interesting pod/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 10:49:30.890840 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.883191    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 10:49:30.928942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.927987    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5wh\" (UniqueName: \"kubernetes.io/projected/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-kube-api-access-fg5wh\") pod \"payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp\" (UID: \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:30.928942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:30.928354    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp\" (UID: \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:31.033062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.032968    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp\" (UID: \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:31.034687 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.034662    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5wh\" (UniqueName: \"kubernetes.io/projected/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-kube-api-access-fg5wh\") pod \"payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp\" (UID: \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:31.035439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.034390    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp\" (UID: \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:31.101991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.101960    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5wh\" (UniqueName: \"kubernetes.io/projected/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-kube-api-access-fg5wh\") pod \"payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp\" (UID: \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:31.135355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.135317    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:49:31.135627 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.135593    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 10:49:31.319605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.319190    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:31.809062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.809015    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:49:31.863434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.863395    5393 patch_prober.go:29] interesting pod/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 10:49:31.863714 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:31.863687    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 10:49:32.066028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:32.064414    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"]
Nov 03 10:49:32.074979 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:32.073748    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077cee3e-zj595"]
Nov 03 10:49:32.147179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:32.145185    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:49:32.147179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:32.145251    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 10:49:32.446564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:32.446325    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3102e53e-32ab-4b1b-9341-5f6faf806cc3" path="/var/lib/kubelet/pods/3102e53e-32ab-4b1b-9341-5f6faf806cc3/volumes"
Nov 03 10:49:32.446788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:32.446768    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"]
Nov 03 10:49:32.833414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:32.833319    5393 patch_prober.go:29] interesting pod/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 10:49:32.833686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:32.833657    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 10:49:33.125590 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:33.116693    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp" event={"ID":"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778","Type":"ContainerStarted","Data":"ffdb6f1d59c52d54ab12a5b6dfe4e67e82769e485902be7848453a3bc5b51551"}
Nov 03 10:49:33.135941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:33.132324    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:49:33.135941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:33.132380    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 10:49:33.825949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:33.825030    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:49:33.831944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:33.830328    5393 patch_prober.go:29] interesting pod/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 10:49:33.831944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:33.830383    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 10:49:34.136729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:34.136604    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:49:34.136729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:34.136665    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 10:49:34.829819 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:34.829348    5393 patch_prober.go:29] interesting pod/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 10:49:34.829819 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:34.829413    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 10:49:35.138731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:35.133262    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:49:35.138731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:35.133324    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 10:49:35.834942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:35.832350    5393 patch_prober.go:29] interesting pod/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 10:49:35.834942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:35.832409    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 10:49:36.135376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:36.135330    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:49:36.135583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:36.135401    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 10:49:36.452286 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:36.451158    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:49:36.835487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:36.835397    5393 patch_prober.go:29] interesting pod/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 10:49:36.835746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:36.835715    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 10:49:37.174452 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:37.173869    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 10:49:37.174452 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:37.173948    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 10:49:37.975210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:37.973616    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:49:37.980999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:37.980460    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:49:38.142255 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:38.142221    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:49:38.144159 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:38.144136    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:49:39.327364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.327323    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp" event={"ID":"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778","Type":"ContainerStarted","Data":"3291347c887fbe85ed275e5f0fc3b1a5a012b1cbb4c9e077e152983000d913be"}
Nov 03 10:49:39.328136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.327537    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:39.342389 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.342326    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp" podStartSLOduration=2.943777964 podStartE2EDuration="9.342307903s" totalImagesPullingTime="6.398529939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:39.342004846 +0000 UTC m=+1223.246859290" watchObservedRunningTime="2026-11-03 10:49:39.342307903 +0000 UTC m=+1223.247162346"
Nov 03 10:49:39.638001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.637961    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"]
Nov 03 10:49:39.642884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.642861    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.650755 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.650240    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-aa420c-serving-cert\""
Nov 03 10:49:39.661581 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.658189    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"]
Nov 03 10:49:39.744452 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.744418    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.744625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.744475    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.744625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.744563    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.744723 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.744705    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlnp\" (UniqueName: \"kubernetes.io/projected/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-kube-api-access-ltlnp\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.846940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.845946    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlnp\" (UniqueName: \"kubernetes.io/projected/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-kube-api-access-ltlnp\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.846940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.846222    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.846940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.846276    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.846940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.846319    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.853459 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:39.847414    5393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-serving-cert: secret "maas-api-e2e-ipp-b-aa420c-serving-cert" not found
Nov 03 10:49:39.853459 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:39.847503    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls podName:6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b nodeName:}" failed. No retries permitted until 2026-11-03 10:49:40.347479755 +0000 UTC m=+1224.252334191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls") pod "maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx" (UID: "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b") : secret "maas-api-e2e-ipp-b-aa420c-serving-cert" not found
Nov 03 10:49:39.860068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.857279    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.866762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.862748    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:39.867770 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:39.867726    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlnp\" (UniqueName: \"kubernetes.io/projected/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-kube-api-access-ltlnp\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:40.354063 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:40.354027    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:40.356797 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:40.356769    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:40.579945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:40.576903    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:40.889080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:40.889050    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"]
Nov 03 10:49:40.889220 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:49:40.889145    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e21a02c_c9d8_4ab2_b689_f2b16bf4d03b.slice/crio-dd600e80606c6ef1984cbda8765e8279d179f17e1fdefe154029310109d6b486 WatchSource:0}: Error finding container dd600e80606c6ef1984cbda8765e8279d179f17e1fdefe154029310109d6b486: Status 404 returned error can't find the container with id dd600e80606c6ef1984cbda8765e8279d179f17e1fdefe154029310109d6b486
Nov 03 10:49:41.400461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:41.400422    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx" event={"ID":"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b","Type":"ContainerStarted","Data":"dd600e80606c6ef1984cbda8765e8279d179f17e1fdefe154029310109d6b486"}
Nov 03 10:49:42.409310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:42.409272    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx" event={"ID":"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b","Type":"ContainerStarted","Data":"0521e06f79f4fe6a1f01d67d05c3359a4cfd0d107449a80dbea0c56db6423dd4"}
Nov 03 10:49:42.409734 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:42.409341    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:42.430589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:42.428641    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx" podStartSLOduration=2.940606034 podStartE2EDuration="3.428622352s" totalImagesPullingTime="488.016318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:42.425566012 +0000 UTC m=+1226.330420454" watchObservedRunningTime="2026-11-03 10:49:42.428622352 +0000 UTC m=+1226.333476795"
Nov 03 10:49:43.191621 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:43.191580    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:49:47.785454 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:47.785421    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"]
Nov 03 10:49:47.786283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:47.786223    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c" podUID="100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" containerName="maas-api" containerID="cri-o://1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286" gracePeriod=30
Nov 03 10:49:48.403947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.396352    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:49:48.444831 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.441154    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:49:48.546008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.526542    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-maas-api-tls\") pod \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") "
Nov 03 10:49:48.546008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.526665    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-metrics-certs\") pod \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") "
Nov 03 10:49:48.546008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.526769    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-openshift-service-ca\") pod \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") "
Nov 03 10:49:48.546008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.526970    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-kube-api-access-l2lr8\" (UniqueName: \"kubernetes.io/projected/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-kube-api-access-l2lr8\") pod \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\" (UID: \"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab\") "
Nov 03 10:49:48.547702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.547602    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-openshift-service-ca" pod "100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" (UID: "100f9b59-5a15-4084-94bf-d9fe0cf1c6ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:48.551504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.551454    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-kube-api-access-l2lr8" pod "100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" (UID: "100f9b59-5a15-4084-94bf-d9fe0cf1c6ab"). InnerVolumeSpecName "kube-api-access-l2lr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:48.576443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.565520    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-maas-api-tls" pod "100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" (UID: "100f9b59-5a15-4084-94bf-d9fe0cf1c6ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:49:48.578039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.577564    5393 generic.go:356] "Generic (PLEG): container finished" podID="100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" containerID="1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286" exitCode=0
Nov 03 10:49:48.578039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.577794    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c" event={"ID":"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab","Type":"ContainerDied","Data":"1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286"}
Nov 03 10:49:48.578039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.577845    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c" event={"ID":"100f9b59-5a15-4084-94bf-d9fe0cf1c6ab","Type":"ContainerDied","Data":"5d50075f875c069dc2ea53383ff95fad3fa9fee595ae63f08674635a208c0100"}
Nov 03 10:49:48.578039 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.577873    5393 scope.go:122] "RemoveContainer" containerID="1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286"
Nov 03 10:49:48.578727 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.578709    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"
Nov 03 10:49:48.583005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.582858    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-metrics-certs" pod "100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" (UID: "100f9b59-5a15-4084-94bf-d9fe0cf1c6ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:49:48.611185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.608844    5393 scope.go:122] "RemoveContainer" containerID="1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286"
Nov 03 10:49:48.611185 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:48.609521    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286\": container with ID starting with 1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286 not found: ID does not exist" containerID="1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286"
Nov 03 10:49:48.611185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.609560    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286"} err="failed to get container status \"1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286\": rpc error: code = NotFound desc = could not find container \"1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286\": container with ID starting with 1adaf59b898100df12cb1bd1c179818cda23042d6b01d0c9089314e7d720e286 not found: ID does not exist"
Nov 03 10:49:48.630174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.630144    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2lr8\" (UniqueName: \"kubernetes.io/projected/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-kube-api-access-l2lr8\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:48.630418 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.630387    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:48.630525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.630514    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:48.630615 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.630605    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:48.635273 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.635245    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"]
Nov 03 10:49:48.635825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.635797    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy" containerID="cri-o://3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5" gracePeriod=30
Nov 03 10:49:48.640879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.639783    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"]
Nov 03 10:49:48.640879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.640050    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy" containerID="cri-o://7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46" gracePeriod=30
Nov 03 10:49:48.657097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.657062    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"]
Nov 03 10:49:48.657246 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.657143    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"]
Nov 03 10:49:48.657346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.657320    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy" containerID="cri-o://5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6" gracePeriod=30
Nov 03 10:49:48.657676 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.657605    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy" containerID="cri-o://6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb" gracePeriod=30
Nov 03 10:49:48.904025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.903994    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"]
Nov 03 10:49:48.906744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:48.906666    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-90bad4-7c88b8c6d9-8mc9c"]
Nov 03 10:49:50.271944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:50.268473    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" path="/var/lib/kubelet/pods/100f9b59-5a15-4084-94bf-d9fe0cf1c6ab/volumes"
Nov 03 10:49:50.366945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:50.364240    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:49:51.354284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.354245    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"]
Nov 03 10:49:51.355298 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.355271    5393 cpu_manager.go:414] "containerMap: removing container" podUID="100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" containerName="maas-api"
Nov 03 10:49:51.355489 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.355463    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" containerName="maas-api"
Nov 03 10:49:51.355709 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.355681    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="100f9b59-5a15-4084-94bf-d9fe0cf1c6ab" containerName="maas-api"
Nov 03 10:49:51.361397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.361369    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:51.367975 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.367950    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"]
Nov 03 10:49:51.372848 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.372824    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4d9549\""
Nov 03 10:49:51.374191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.374167    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4d9549-dockercfg-przpl\""
Nov 03 10:49:51.590324 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.587349    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr8x9\" (UniqueName: \"kubernetes.io/projected/d451027a-a892-4f68-9da5-79ea8eab7d84-kube-api-access-dr8x9\") pod \"payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm\" (UID: \"d451027a-a892-4f68-9da5-79ea8eab7d84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:51.590324 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.587534    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d451027a-a892-4f68-9da5-79ea8eab7d84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm\" (UID: \"d451027a-a892-4f68-9da5-79ea8eab7d84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:51.711070 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.709986    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d451027a-a892-4f68-9da5-79ea8eab7d84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm\" (UID: \"d451027a-a892-4f68-9da5-79ea8eab7d84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:51.711070 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.710377    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8x9\" (UniqueName: \"kubernetes.io/projected/d451027a-a892-4f68-9da5-79ea8eab7d84-kube-api-access-dr8x9\") pod \"payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm\" (UID: \"d451027a-a892-4f68-9da5-79ea8eab7d84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:51.715570 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.711820    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d451027a-a892-4f68-9da5-79ea8eab7d84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm\" (UID: \"d451027a-a892-4f68-9da5-79ea8eab7d84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:51.729173 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.729144    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8x9\" (UniqueName: \"kubernetes.io/projected/d451027a-a892-4f68-9da5-79ea8eab7d84-kube-api-access-dr8x9\") pod \"payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm\" (UID: \"d451027a-a892-4f68-9da5-79ea8eab7d84\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:51.984003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:51.982560    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:52.321094 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:52.321053    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"]
Nov 03 10:49:52.322244 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:49:52.322213    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd451027a_a892_4f68_9da5_79ea8eab7d84.slice/crio-370011d3ed244a9b394685a2f506aa894074a386efe0264ccf72c86c76e4b226 WatchSource:0}: Error finding container 370011d3ed244a9b394685a2f506aa894074a386efe0264ccf72c86c76e4b226: Status 404 returned error can't find the container with id 370011d3ed244a9b394685a2f506aa894074a386efe0264ccf72c86c76e4b226
Nov 03 10:49:52.694522 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:52.694477    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm" event={"ID":"d451027a-a892-4f68-9da5-79ea8eab7d84","Type":"ContainerStarted","Data":"370011d3ed244a9b394685a2f506aa894074a386efe0264ccf72c86c76e4b226"}
Nov 03 10:49:53.542228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:53.542174    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-rh242 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:49:53.542422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:53.542241    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:49:53.693638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:53.693603    5393 patch_prober.go:29] interesting pod/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 03 10:49:53.693808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:53.693668    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 03 10:49:53.969336 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:53.969307    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:49:54.061488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061384    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-credential-socket\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.061585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061488    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-socket\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.061585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061574    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-token\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-token\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.061700 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061604    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-data\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.061700 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061674    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3270be3d-f327-4ba8-862f-e88bdf0583f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3270be3d-f327-4ba8-862f-e88bdf0583f1-istiod-ca-cert\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.061801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061730    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-envoy\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.061801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061784    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-kube-api-access-qzwfk\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-kube-api-access-qzwfk\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.061948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061844    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-podinfo\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.061948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.061911    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-certs\") pod \"3270be3d-f327-4ba8-862f-e88bdf0583f1\" (UID: \"3270be3d-f327-4ba8-862f-e88bdf0583f1\") "
Nov 03 10:49:54.062906 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.062867    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-certs" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.063342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.063171    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-credential-socket" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.065383 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.063799    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-socket" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.066080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.065783    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-data" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.069332 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.066529    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3270be3d-f327-4ba8-862f-e88bdf0583f1-istiod-ca-cert" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:54.071049 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.071008    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-envoy" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.079775 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.079667    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:49:54.090765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.090685    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-token" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.092133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.092087    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-podinfo" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:49:54.094276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.094249    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-kube-api-access-qzwfk" pod "3270be3d-f327-4ba8-862f-e88bdf0583f1" (UID: "3270be3d-f327-4ba8-862f-e88bdf0583f1"). InnerVolumeSpecName "kube-api-access-qzwfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.109262 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.109193    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.163992    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-data\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164039    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-podinfo\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164094    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-credential-socket\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164120    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-token\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-token\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164159    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-certs\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164224    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-kube-api-access-9bzcs\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164264    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b77b64a-1da8-45e6-825c-81dee0a0366a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77b64a-1da8-45e6-825c-81dee0a0366a-istiod-ca-cert\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164315    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-socket\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164335    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-data" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164348    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-data\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164499    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-credential-socket\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164588    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-token\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-token\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164661    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-kube-api-access-8vcw9\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-kube-api-access-8vcw9\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164700    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-podinfo\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164742    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-envoy\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164802    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-certs\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164847    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-socket\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.164940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164889    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fed644-0876-44be-ad2c-4ff36b228ca6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fed644-0876-44be-ad2c-4ff36b228ca6-istiod-ca-cert\") pod \"a8fed644-0876-44be-ad2c-4ff36b228ca6\" (UID: \"a8fed644-0876-44be-ad2c-4ff36b228ca6\") "
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.164992    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-envoy\") pod \"8b77b64a-1da8-45e6-825c-81dee0a0366a\" (UID: \"8b77b64a-1da8-45e6-825c-81dee0a0366a\") "
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.165977    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-socket" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166027    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-certs" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166436    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b77b64a-1da8-45e6-825c-81dee0a0366a-istiod-ca-cert" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166601    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwfk\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-kube-api-access-qzwfk\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166623    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166638    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166652    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166666    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166678    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166694    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166707    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166720    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b77b64a-1da8-45e6-825c-81dee0a0366a-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.167749    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.167768    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.167776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.167784    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3270be3d-f327-4ba8-862f-e88bdf0583f1-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.168629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.167799    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3270be3d-f327-4ba8-862f-e88bdf0583f1-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.168629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166935    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fed644-0876-44be-ad2c-4ff36b228ca6-istiod-ca-cert" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:54.168629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.166973    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-credential-socket" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.168629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.167009    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-certs" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.168629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.167457    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-credential-socket" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.168629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.167515    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-socket" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.168629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.167897    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-data" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.177731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.177645    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-token" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.178013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.177938    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-envoy" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.178804 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.178694    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-envoy" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.178804 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.178761    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-kube-api-access-8vcw9" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "kube-api-access-8vcw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.180521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.180493    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-token" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.180624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.180527    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-podinfo" pod "a8fed644-0876-44be-ad2c-4ff36b228ca6" (UID: "a8fed644-0876-44be-ad2c-4ff36b228ca6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:49:54.180624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.180556    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-kube-api-access-9bzcs" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "kube-api-access-9bzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.187034 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.182793    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-podinfo" pod "8b77b64a-1da8-45e6-825c-81dee0a0366a" (UID: "8b77b64a-1da8-45e6-825c-81dee0a0366a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:49:54.241620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.241584    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"]
Nov 03 10:49:54.243777 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.243748    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy"
Nov 03 10:49:54.243777 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.243775    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy"
Nov 03 10:49:54.244114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.243813    5393 cpu_manager.go:414] "containerMap: removing container" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy"
Nov 03 10:49:54.244114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.243821    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy"
Nov 03 10:49:54.244114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.243844    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy"
Nov 03 10:49:54.244114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.243850    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy"
Nov 03 10:49:54.244114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.244023    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerName="istio-proxy"
Nov 03 10:49:54.244114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.244041    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerName="istio-proxy"
Nov 03 10:49:54.244114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.244053    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerName="istio-proxy"
Nov 03 10:49:54.249570 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.249547    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.263236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.263209    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a2697c-openshift-default-dockercfg-88xzv\""
Nov 03 10:49:54.270499 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.270463    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-data\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.270684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.270661    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-credential-socket\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.271050 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.271028    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltz46\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-kube-api-access-ltz46\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.271197 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.271179    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-envoy\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.271347 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.271331    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istiod-ca-cert\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.271497 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.271482    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-socket\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.271625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.271610    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-token\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.271758 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.271741    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-podinfo\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.271929 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.271898    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-certs\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.272281 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.272263    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.272483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.272467    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.272625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.272606    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.272732 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.272718    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzcs\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-kube-api-access-9bzcs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.272825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.272813    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.272934 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.272902    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.273030 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.273017    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.273138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.273125    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.276504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.273216    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vcw9\" (UniqueName: \"kubernetes.io/projected/a8fed644-0876-44be-ad2c-4ff36b228ca6-kube-api-access-8vcw9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.276653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.276636    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.276745 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.276733    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fed644-0876-44be-ad2c-4ff36b228ca6-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.276833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.276821    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.276936 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.276904    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fed644-0876-44be-ad2c-4ff36b228ca6-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.277030 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.277018    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b77b64a-1da8-45e6-825c-81dee0a0366a-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.284564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.284526    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"]
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.378800    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-data\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.378867    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-credential-socket\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.379057    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltz46\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-kube-api-access-ltz46\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.379087    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-envoy\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.379132    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istiod-ca-cert\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.379185    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-socket\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.379227    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-token\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.379267    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-podinfo\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.379322    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-certs\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.379768    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-certs\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.380009    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-data\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.380391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.380259    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-socket\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.381286 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.381229    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-credential-socket\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.381543 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.381513    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istiod-ca-cert\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.384158 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.384130    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-envoy\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.384343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.384319    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-podinfo\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.389025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.388998    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltz46\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-kube-api-access-ltz46\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.389160 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.389139    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-token\") pod \"e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") " pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.423619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.423591    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.486404    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-envoy\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.486528    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-credential-socket\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.486557    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-data\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.486611    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133865c0-b83e-4246-95ec-731579037dd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133865c0-b83e-4246-95ec-731579037dd5-istiod-ca-cert\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.486713    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-kube-api-access-pm4c8\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-kube-api-access-pm4c8\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.486812    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/133865c0-b83e-4246-95ec-731579037dd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133865c0-b83e-4246-95ec-731579037dd5-istio-podinfo\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.486946    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-istio-token\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-istio-token\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.487033    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-socket\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.492663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.487089    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-certs\") pod \"133865c0-b83e-4246-95ec-731579037dd5\" (UID: \"133865c0-b83e-4246-95ec-731579037dd5\") "
Nov 03 10:49:54.495071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.495029    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-certs" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.514038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.513730    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-data" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.515474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.515241    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-socket" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.515474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.515432    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133865c0-b83e-4246-95ec-731579037dd5-istiod-ca-cert" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:54.515847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.515804    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-credential-socket" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.554124 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.554046    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-envoy" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:49:54.565889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.565831    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-kube-api-access-pm4c8" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "kube-api-access-pm4c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.578350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.578288    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/133865c0-b83e-4246-95ec-731579037dd5-istio-podinfo" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:49:54.584368 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.584314    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-istio-token" pod "133865c0-b83e-4246-95ec-731579037dd5" (UID: "133865c0-b83e-4246-95ec-731579037dd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605503    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/133865c0-b83e-4246-95ec-731579037dd5-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605541    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm4c8\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-kube-api-access-pm4c8\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605556    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/133865c0-b83e-4246-95ec-731579037dd5-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605570    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/133865c0-b83e-4246-95ec-731579037dd5-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605586    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605599    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605613    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605629    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.616525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.605643    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/133865c0-b83e-4246-95ec-731579037dd5-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:54.617941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.617588    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:54.622540 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.622512    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"]
Nov 03 10:49:54.822952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.820497    5393 generic.go:356] "Generic (PLEG): container finished" podID="3270be3d-f327-4ba8-862f-e88bdf0583f1" containerID="3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5" exitCode=0
Nov 03 10:49:54.822952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.820661    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" event={"ID":"3270be3d-f327-4ba8-862f-e88bdf0583f1","Type":"ContainerDied","Data":"3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5"}
Nov 03 10:49:54.822952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.820722    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w" event={"ID":"3270be3d-f327-4ba8-862f-e88bdf0583f1","Type":"ContainerDied","Data":"c9f9eef2efd6dfaf7eabb6d103722e679ae970c60d12ec7e7eb45836723fb6ec"}
Nov 03 10:49:54.822952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.820754    5393 scope.go:122] "RemoveContainer" containerID="3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5"
Nov 03 10:49:54.822952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.820975    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"
Nov 03 10:49:54.827891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.825356    5393 generic.go:356] "Generic (PLEG): container finished" podID="8b77b64a-1da8-45e6-825c-81dee0a0366a" containerID="5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6" exitCode=0
Nov 03 10:49:54.827891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.826624    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"
Nov 03 10:49:54.827891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.827339    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" event={"ID":"8b77b64a-1da8-45e6-825c-81dee0a0366a","Type":"ContainerDied","Data":"5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6"}
Nov 03 10:49:54.827891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.827393    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8" event={"ID":"8b77b64a-1da8-45e6-825c-81dee0a0366a","Type":"ContainerDied","Data":"7aabc6e7c7e6f88fbc856bd90811e467aa1d80da0ac132c10a5dd85de05b3bd5"}
Nov 03 10:49:54.832259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.832050    5393 generic.go:356] "Generic (PLEG): container finished" podID="a8fed644-0876-44be-ad2c-4ff36b228ca6" containerID="6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb" exitCode=0
Nov 03 10:49:54.832259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.832100    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"
Nov 03 10:49:54.832259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.832148    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" event={"ID":"a8fed644-0876-44be-ad2c-4ff36b228ca6","Type":"ContainerDied","Data":"6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb"}
Nov 03 10:49:54.832259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.832186    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242" event={"ID":"a8fed644-0876-44be-ad2c-4ff36b228ca6","Type":"ContainerDied","Data":"c97ebf6b5729c7cdeee5c5165c951851f1659c85fbfbdfcc087a272b9c8592eb"}
Nov 03 10:49:54.835394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.835246    5393 generic.go:356] "Generic (PLEG): container finished" podID="133865c0-b83e-4246-95ec-731579037dd5" containerID="7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46" exitCode=0
Nov 03 10:49:54.835394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.835334    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" event={"ID":"133865c0-b83e-4246-95ec-731579037dd5","Type":"ContainerDied","Data":"7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46"}
Nov 03 10:49:54.835394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.835378    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4" event={"ID":"133865c0-b83e-4246-95ec-731579037dd5","Type":"ContainerDied","Data":"d172020948f75627da38c4a4aa9017ffbe1ceb42310e08593b24e8aa1cd77488"}
Nov 03 10:49:54.835696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.835420    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"
Nov 03 10:49:54.886055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.885456    5393 scope.go:122] "RemoveContainer" containerID="3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5"
Nov 03 10:49:54.887407 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:54.886639    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5\": container with ID starting with 3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5 not found: ID does not exist" containerID="3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5"
Nov 03 10:49:54.887407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.886684    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5"} err="failed to get container status \"3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5\": rpc error: code = NotFound desc = could not find container \"3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5\": container with ID starting with 3e902a2136b7d071fcbd0cab85e0c4fe4173358573b2aa9d21b8d294ef2a75d5 not found: ID does not exist"
Nov 03 10:49:54.887407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.886707    5393 scope.go:122] "RemoveContainer" containerID="5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6"
Nov 03 10:49:54.893626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.893422    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"]
Nov 03 10:49:54.896765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.896731    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-p75w4"]
Nov 03 10:49:54.916073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.915905    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"]
Nov 03 10:49:54.923874 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.923824    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-wjm9w"]
Nov 03 10:49:54.934717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.934657    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"]
Nov 03 10:49:54.945322 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.944979    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-552v8"]
Nov 03 10:49:54.945627 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.945516    5393 scope.go:122] "RemoveContainer" containerID="5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6"
Nov 03 10:49:54.952225 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:54.952050    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6\": container with ID starting with 5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6 not found: ID does not exist" containerID="5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6"
Nov 03 10:49:54.952225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.952095    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6"} err="failed to get container status \"5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6\": rpc error: code = NotFound desc = could not find container \"5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6\": container with ID starting with 5d34be6524cb132a4ad872e7980582938b645695be39b66b4fb7750ea0232eb6 not found: ID does not exist"
Nov 03 10:49:54.952225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.952120    5393 scope.go:122] "RemoveContainer" containerID="6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb"
Nov 03 10:49:54.973652 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.973627    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"]
Nov 03 10:49:54.974704 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:54.974684    5393 scope.go:122] "RemoveContainer" containerID="6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb"
Nov 03 10:49:55.094942 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:55.088589    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb\": container with ID starting with 6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb not found: ID does not exist" containerID="6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb"
Nov 03 10:49:55.094942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.088636    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb"} err="failed to get container status \"6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb\": rpc error: code = NotFound desc = could not find container \"6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb\": container with ID starting with 6f50fad7ce1330475c78ffcf8081aba165e49e9224381f7f8ca2fcfbabc64ddb not found: ID does not exist"
Nov 03 10:49:55.094942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.088665    5393 scope.go:122] "RemoveContainer" containerID="7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46"
Nov 03 10:49:55.211946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.209501    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab592834-openshift-default-7f97f8769-rh242"]
Nov 03 10:49:55.443857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.443819    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"]
Nov 03 10:49:55.445742 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.445720    5393 scope.go:122] "RemoveContainer" containerID="7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46"
Nov 03 10:49:55.455134 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:49:55.455090    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46\": container with ID starting with 7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46 not found: ID does not exist" containerID="7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46"
Nov 03 10:49:55.455240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.455158    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46"} err="failed to get container status \"7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46\": rpc error: code = NotFound desc = could not find container \"7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46\": container with ID starting with 7ac410eefca13d343dda5f004308fe5647f1cb134d90391785649f1463bedb46 not found: ID does not exist"
Nov 03 10:49:55.459434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.459409    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"]
Nov 03 10:49:55.459855 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.459825    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy" containerID="cri-o://c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0" gracePeriod=30
Nov 03 10:49:55.469463 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.469415    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:55.469580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.469560    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:55.469669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.469652    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:49:55.997583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.997546    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" event={"ID":"ce655ad0-1a5a-4615-863a-9fd43685b4bb","Type":"ContainerStarted","Data":"287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0"}
Nov 03 10:49:55.998069 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:55.997608    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" event={"ID":"ce655ad0-1a5a-4615-863a-9fd43685b4bb","Type":"ContainerStarted","Data":"b8cfa78174219bc112e1f99395b4db5dc05ea442e1ed5fb17b853d3a65cbedaf"}
Nov 03 10:49:56.025672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.025574    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" podStartSLOduration=2.025554649 podStartE2EDuration="2.025554649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:56.024428709 +0000 UTC m=+1239.929283150" watchObservedRunningTime="2026-11-03 10:49:56.025554649 +0000 UTC m=+1239.930409090"
Nov 03 10:49:56.043957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.042999    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"]
Nov 03 10:49:56.043957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.043888    5393 cpu_manager.go:414] "containerMap: removing container" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy"
Nov 03 10:49:56.043957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.043907    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy"
Nov 03 10:49:56.044623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.044119    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="133865c0-b83e-4246-95ec-731579037dd5" containerName="istio-proxy"
Nov 03 10:49:56.049516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.049405    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"]
Nov 03 10:49:56.049654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.049523    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.053338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.053314    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:49:56.054707 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.054593    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:49:56.220335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.220295    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4388dcef-532e-4854-9846-1a5701465758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.229155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.229122    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4388dcef-532e-4854-9846-1a5701465758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.229466 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.229445    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdqz\" (UniqueName: \"kubernetes.io/projected/4388dcef-532e-4854-9846-1a5701465758-kube-api-access-kpdqz\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.262817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.262443    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133865c0-b83e-4246-95ec-731579037dd5" path="/var/lib/kubelet/pods/133865c0-b83e-4246-95ec-731579037dd5/volumes"
Nov 03 10:49:56.265757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.263193    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3270be3d-f327-4ba8-862f-e88bdf0583f1" path="/var/lib/kubelet/pods/3270be3d-f327-4ba8-862f-e88bdf0583f1/volumes"
Nov 03 10:49:56.265757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.263850    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b77b64a-1da8-45e6-825c-81dee0a0366a" path="/var/lib/kubelet/pods/8b77b64a-1da8-45e6-825c-81dee0a0366a/volumes"
Nov 03 10:49:56.265757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.264874    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fed644-0876-44be-ad2c-4ff36b228ca6" path="/var/lib/kubelet/pods/a8fed644-0876-44be-ad2c-4ff36b228ca6/volumes"
Nov 03 10:49:56.330825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.330790    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4388dcef-532e-4854-9846-1a5701465758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.331041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.330857    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4388dcef-532e-4854-9846-1a5701465758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.331041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.330969    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdqz\" (UniqueName: \"kubernetes.io/projected/4388dcef-532e-4854-9846-1a5701465758-kube-api-access-kpdqz\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.331795 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.331767    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4388dcef-532e-4854-9846-1a5701465758-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.334597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.334562    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4388dcef-532e-4854-9846-1a5701465758-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.341469 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.341443    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdqz\" (UniqueName: \"kubernetes.io/projected/4388dcef-532e-4854-9846-1a5701465758-kube-api-access-kpdqz\") pod \"maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.372256 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.372221    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:56.618550 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.618511    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:56.624103 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.624070    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:56.674066 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:56.674039    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"]
Nov 03 10:49:56.676978 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:49:56.676948    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4388dcef_532e_4854_9846_1a5701465758.slice/crio-9b20ba42a1bd8a09227792a8502c4e2e7cf54d5383739af31dcb11373e14f40e WatchSource:0}: Error finding container 9b20ba42a1bd8a09227792a8502c4e2e7cf54d5383739af31dcb11373e14f40e: Status 404 returned error can't find the container with id 9b20ba42a1bd8a09227792a8502c4e2e7cf54d5383739af31dcb11373e14f40e
Nov 03 10:49:57.012011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:57.011973    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5" event={"ID":"4388dcef-532e-4854-9846-1a5701465758","Type":"ContainerStarted","Data":"9b20ba42a1bd8a09227792a8502c4e2e7cf54d5383739af31dcb11373e14f40e"}
Nov 03 10:49:57.013998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:57.013957    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm" event={"ID":"d451027a-a892-4f68-9da5-79ea8eab7d84","Type":"ContainerStarted","Data":"19e4450a8552162c48498fc3e99898cd1f6a3f33bffd03e4d5011618d95d4a35"}
Nov 03 10:49:57.014335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:57.014313    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:57.014458 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:57.014359    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:49:57.015454 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:57.015430    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:49:57.033236 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:57.033180    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm" podStartSLOduration=2.651336087 podStartE2EDuration="6.033164393s" totalImagesPullingTime="3.381828306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:49:57.03018453 +0000 UTC m=+1240.935038972" watchObservedRunningTime="2026-11-03 10:49:57.033164393 +0000 UTC m=+1240.938018833"
Nov 03 10:49:58.023510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:58.023480    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"]
Nov 03 10:49:58.024320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:58.024299    5393 generic.go:356] "Generic (PLEG): container finished" podID="4388dcef-532e-4854-9846-1a5701465758" containerID="abaae3c7bf1e56f8d9f0b183252c53933e9943ebc2584537e4517188b85dde03" exitCode=0
Nov 03 10:49:58.024419 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:58.024392    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5" event={"ID":"4388dcef-532e-4854-9846-1a5701465758","Type":"ContainerDied","Data":"abaae3c7bf1e56f8d9f0b183252c53933e9943ebc2584537e4517188b85dde03"}
Nov 03 10:49:58.027180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:58.027158    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6ebd9d-khmg6"]
Nov 03 10:49:58.242987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:58.242900    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9979c146-a608-4130-9ad4-ad216056b5a9" path="/var/lib/kubelet/pods/9979c146-a608-4130-9ad4-ad216056b5a9/volumes"
Nov 03 10:49:59.204809 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.204786    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:49:59.309327 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.309252    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4388dcef-532e-4854-9846-1a5701465758-kube-api-access-kpdqz\" (UniqueName: \"kubernetes.io/projected/4388dcef-532e-4854-9846-1a5701465758-kube-api-access-kpdqz\") pod \"4388dcef-532e-4854-9846-1a5701465758\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") "
Nov 03 10:49:59.309475 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.309363    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4388dcef-532e-4854-9846-1a5701465758-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4388dcef-532e-4854-9846-1a5701465758-maas-api-xks-ca\") pod \"4388dcef-532e-4854-9846-1a5701465758\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") "
Nov 03 10:49:59.309475 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.309469    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4388dcef-532e-4854-9846-1a5701465758-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4388dcef-532e-4854-9846-1a5701465758-maas-api-openshift-ca\") pod \"4388dcef-532e-4854-9846-1a5701465758\" (UID: \"4388dcef-532e-4854-9846-1a5701465758\") "
Nov 03 10:49:59.309822 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.309796    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4388dcef-532e-4854-9846-1a5701465758-maas-api-openshift-ca" pod "4388dcef-532e-4854-9846-1a5701465758" (UID: "4388dcef-532e-4854-9846-1a5701465758"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:49:59.310170 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.310092    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4388dcef-532e-4854-9846-1a5701465758-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:59.311567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.311541    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4388dcef-532e-4854-9846-1a5701465758-kube-api-access-kpdqz" pod "4388dcef-532e-4854-9846-1a5701465758" (UID: "4388dcef-532e-4854-9846-1a5701465758"). InnerVolumeSpecName "kube-api-access-kpdqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:49:59.311675 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.311639    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4388dcef-532e-4854-9846-1a5701465758-maas-api-xks-ca" pod "4388dcef-532e-4854-9846-1a5701465758" (UID: "4388dcef-532e-4854-9846-1a5701465758"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:49:59.411362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.411327    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4388dcef-532e-4854-9846-1a5701465758-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:59.411362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.411357    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpdqz\" (UniqueName: \"kubernetes.io/projected/4388dcef-532e-4854-9846-1a5701465758-kube-api-access-kpdqz\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:49:59.558091 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.558052    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"]
Nov 03 10:49:59.558638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.558626    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4388dcef-532e-4854-9846-1a5701465758" containerName="revoke-keys"
Nov 03 10:49:59.558701 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.558640    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4388dcef-532e-4854-9846-1a5701465758" containerName="revoke-keys"
Nov 03 10:49:59.558763 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.558755    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4388dcef-532e-4854-9846-1a5701465758" containerName="revoke-keys"
Nov 03 10:49:59.562651 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.562605    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:49:59.569242 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.569179    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a2697c\""
Nov 03 10:49:59.569463 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.569431    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a2697c-dockercfg-ms4rl\""
Nov 03 10:49:59.574110 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.574091    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"]
Nov 03 10:49:59.589889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.589855    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"]
Nov 03 10:49:59.594316 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.594298    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:49:59.601538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.601510    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"]
Nov 03 10:49:59.627893 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.627828    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp" podUID="d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" containerName="payload-processing" containerID="cri-o://3291347c887fbe85ed275e5f0fc3b1a5a012b1cbb4c9e077e152983000d913be" gracePeriod=25
Nov 03 10:49:59.763951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.763172    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f18330-9312-499f-96d7-37f6c77dd1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5\" (UID: \"54f18330-9312-499f-96d7-37f6c77dd1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:49:59.763951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.763241    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvvk\" (UniqueName: \"kubernetes.io/projected/54f18330-9312-499f-96d7-37f6c77dd1a1-kube-api-access-2mvvk\") pod \"payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5\" (UID: \"54f18330-9312-499f-96d7-37f6c77dd1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:49:59.898940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.871020    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded51902-10c7-474c-bb34-a8b97bc86562-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd\" (UID: \"ded51902-10c7-474c-bb34-a8b97bc86562\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:49:59.898940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.871143    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f18330-9312-499f-96d7-37f6c77dd1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5\" (UID: \"54f18330-9312-499f-96d7-37f6c77dd1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:49:59.898940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.871204    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvvk\" (UniqueName: \"kubernetes.io/projected/54f18330-9312-499f-96d7-37f6c77dd1a1-kube-api-access-2mvvk\") pod \"payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5\" (UID: \"54f18330-9312-499f-96d7-37f6c77dd1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:49:59.898940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.871488    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6f9f\" (UniqueName: \"kubernetes.io/projected/ded51902-10c7-474c-bb34-a8b97bc86562-kube-api-access-z6f9f\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd\" (UID: \"ded51902-10c7-474c-bb34-a8b97bc86562\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:49:59.898940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.872940    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f18330-9312-499f-96d7-37f6c77dd1a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5\" (UID: \"54f18330-9312-499f-96d7-37f6c77dd1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:49:59.964715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.949711    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvvk\" (UniqueName: \"kubernetes.io/projected/54f18330-9312-499f-96d7-37f6c77dd1a1-kube-api-access-2mvvk\") pod \"payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5\" (UID: \"54f18330-9312-499f-96d7-37f6c77dd1a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:49:59.992938 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.992385    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6f9f\" (UniqueName: \"kubernetes.io/projected/ded51902-10c7-474c-bb34-a8b97bc86562-kube-api-access-z6f9f\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd\" (UID: \"ded51902-10c7-474c-bb34-a8b97bc86562\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:49:59.992938 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.992446    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded51902-10c7-474c-bb34-a8b97bc86562-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd\" (UID: \"ded51902-10c7-474c-bb34-a8b97bc86562\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:49:59.999766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:49:59.997822    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded51902-10c7-474c-bb34-a8b97bc86562-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd\" (UID: \"ded51902-10c7-474c-bb34-a8b97bc86562\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:50:00.065966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.062800    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6f9f\" (UniqueName: \"kubernetes.io/projected/ded51902-10c7-474c-bb34-a8b97bc86562-kube-api-access-z6f9f\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd\" (UID: \"ded51902-10c7-474c-bb34-a8b97bc86562\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:50:00.098946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.095232    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5" event={"ID":"4388dcef-532e-4854-9846-1a5701465758","Type":"ContainerDied","Data":"9b20ba42a1bd8a09227792a8502c4e2e7cf54d5383739af31dcb11373e14f40e"}
Nov 03 10:50:00.098946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.095295    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b20ba42a1bd8a09227792a8502c4e2e7cf54d5383739af31dcb11373e14f40e"
Nov 03 10:50:00.098946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.095373    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"
Nov 03 10:50:00.104279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.104239    5393 generic.go:356] "Generic (PLEG): container finished" podID="d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" containerID="3291347c887fbe85ed275e5f0fc3b1a5a012b1cbb4c9e077e152983000d913be" exitCode=0
Nov 03 10:50:00.104453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.104417    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp" event={"ID":"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778","Type":"ContainerDied","Data":"3291347c887fbe85ed275e5f0fc3b1a5a012b1cbb4c9e077e152983000d913be"}
Nov 03 10:50:00.178415 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.178329    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:50:00.315306 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.315271    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:50:00.514835 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.514425    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:50:00.587755 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.587725    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-kube-api-access-fg5wh\" (UniqueName: \"kubernetes.io/projected/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-kube-api-access-fg5wh\") pod \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\" (UID: \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\") "
Nov 03 10:50:00.588276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.587904    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-plugins-config-volume\") pod \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\" (UID: \"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778\") "
Nov 03 10:50:00.588276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.588249    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-plugins-config-volume" pod "d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" (UID: "d2b40ca1-4b45-4ab2-8c80-5c7b4215d778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:50:00.588623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.588606    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.590890 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.590870    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-kube-api-access-fg5wh" pod "d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" (UID: "d2b40ca1-4b45-4ab2-8c80-5c7b4215d778"). InnerVolumeSpecName "kube-api-access-fg5wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:00.631505 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.631479    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"]
Nov 03 10:50:00.690653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.690615    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg5wh\" (UniqueName: \"kubernetes.io/projected/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778-kube-api-access-fg5wh\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.741484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.741458    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:50:00.780890 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:50:00.780862    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f18330_9312_499f_96d7_37f6c77dd1a1.slice/crio-ca6ba131eec8bd27cedc1a9a9e271c2ca840065e97633b4fc205f895ef802f7b WatchSource:0}: Error finding container ca6ba131eec8bd27cedc1a9a9e271c2ca840065e97633b4fc205f895ef802f7b: Status 404 returned error can't find the container with id ca6ba131eec8bd27cedc1a9a9e271c2ca840065e97633b4fc205f895ef802f7b
Nov 03 10:50:00.783180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.783147    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"]
Nov 03 10:50:00.791884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.791856    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-socket\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.791992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.791969    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-kube-api-access-wn56f\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-kube-api-access-wn56f\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.792057 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.792042    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-token\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-token\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.792125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.792077    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-envoy\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.792172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.792155    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d4380b-8e6e-4505-a07e-4bf980ec7296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4380b-8e6e-4505-a07e-4bf980ec7296-istiod-ca-cert\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.792230 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.792216    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-credential-socket\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.792275 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.792264    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-certs\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.792320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.792307    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-podinfo\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.792450 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.792436    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-data\") pod \"28d4380b-8e6e-4505-a07e-4bf980ec7296\" (UID: \"28d4380b-8e6e-4505-a07e-4bf980ec7296\") "
Nov 03 10:50:00.793694 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.793572    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-credential-socket" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:00.793694 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.793602    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-data" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:00.793694 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.793650    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-certs" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:00.793896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.793736    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d4380b-8e6e-4505-a07e-4bf980ec7296-istiod-ca-cert" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:50:00.794271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.794150    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-socket" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:00.796116 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.796087    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-envoy" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:00.796711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.796568    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-kube-api-access-wn56f" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "kube-api-access-wn56f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:00.796711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.796696    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-token" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:00.797040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.797014    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-podinfo" pod "28d4380b-8e6e-4505-a07e-4bf980ec7296" (UID: "28d4380b-8e6e-4505-a07e-4bf980ec7296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:50:00.894624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894592    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.894624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894623    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn56f\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-kube-api-access-wn56f\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.894829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894635    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.894829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894643    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.894829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894653    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d4380b-8e6e-4505-a07e-4bf980ec7296-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.894829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894661    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.894829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894669    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.894829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894676    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:00.894829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:00.894685    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d4380b-8e6e-4505-a07e-4bf980ec7296-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:01.145341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.145236    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd" event={"ID":"ded51902-10c7-474c-bb34-a8b97bc86562","Type":"ContainerStarted","Data":"20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8"}
Nov 03 10:50:01.145613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.145545    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd" event={"ID":"ded51902-10c7-474c-bb34-a8b97bc86562","Type":"ContainerStarted","Data":"d81ded4935e45ae4fbc2fc5f1985a4da3a20b47f07d71388f6814356da54224a"}
Nov 03 10:50:01.157037 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.156734    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:50:01.225270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.221731    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp" event={"ID":"d2b40ca1-4b45-4ab2-8c80-5c7b4215d778","Type":"ContainerDied","Data":"ffdb6f1d59c52d54ab12a5b6dfe4e67e82769e485902be7848453a3bc5b51551"}
Nov 03 10:50:01.225270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.221803    5393 scope.go:122] "RemoveContainer" containerID="3291347c887fbe85ed275e5f0fc3b1a5a012b1cbb4c9e077e152983000d913be"
Nov 03 10:50:01.225270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.222027    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"
Nov 03 10:50:01.257391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.246028    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5" event={"ID":"54f18330-9312-499f-96d7-37f6c77dd1a1","Type":"ContainerStarted","Data":"05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175"}
Nov 03 10:50:01.257391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.246085    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5" event={"ID":"54f18330-9312-499f-96d7-37f6c77dd1a1","Type":"ContainerStarted","Data":"ca6ba131eec8bd27cedc1a9a9e271c2ca840065e97633b4fc205f895ef802f7b"}
Nov 03 10:50:01.257391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.246711    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:50:01.257391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.250214    5393 generic.go:356] "Generic (PLEG): container finished" podID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerID="c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0" exitCode=0
Nov 03 10:50:01.257391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.250284    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" event={"ID":"28d4380b-8e6e-4505-a07e-4bf980ec7296","Type":"ContainerDied","Data":"c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0"}
Nov 03 10:50:01.257391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.250338    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th" event={"ID":"28d4380b-8e6e-4505-a07e-4bf980ec7296","Type":"ContainerDied","Data":"e111c03c45de25753f79321a87cc6c2fb3d51b266b9e7d514494d46699890989"}
Nov 03 10:50:01.257391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.250426    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"
Nov 03 10:50:01.279352 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.279268    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd" podStartSLOduration=2.279245954 podStartE2EDuration="2.279245954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:01.242072579 +0000 UTC m=+1245.146927021" watchObservedRunningTime="2026-11-03 10:50:01.279245954 +0000 UTC m=+1245.184100396"
Nov 03 10:50:01.297341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.293101    5393 scope.go:122] "RemoveContainer" containerID="c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0"
Nov 03 10:50:01.341163 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.341120    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp" podUID="d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.102:50052\" within 1s: context deadline exceeded"
Nov 03 10:50:01.342093 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:50:01.342073    5393 logging.go:55] [core] [Channel #26 SubChannel #27]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.102:50052", ServerName: "10.134.0.102:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.102:50052: operation was canceled"
Nov 03 10:50:01.346535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.345423    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5" podStartSLOduration=2.345404741 podStartE2EDuration="2.345404741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:01.268709627 +0000 UTC m=+1245.173564070" watchObservedRunningTime="2026-11-03 10:50:01.345404741 +0000 UTC m=+1245.250259183"
Nov 03 10:50:01.375781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.375376    5393 scope.go:122] "RemoveContainer" containerID="ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e"
Nov 03 10:50:01.377653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.377637    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"]
Nov 03 10:50:01.400093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.400010    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-90bad4-openshift-default-7b77796d5c-df9th"]
Nov 03 10:50:01.455601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.455479    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"]
Nov 03 10:50:01.488512 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.474765    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa420c-59dd6674d5-ng2mp"]
Nov 03 10:50:01.490794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.490629    5393 scope.go:122] "RemoveContainer" containerID="c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0"
Nov 03 10:50:01.491431 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:01.491290    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0\": container with ID starting with c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0 not found: ID does not exist" containerID="c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0"
Nov 03 10:50:01.491431 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.491328    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0"} err="failed to get container status \"c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0\": rpc error: code = NotFound desc = could not find container \"c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0\": container with ID starting with c802ddecb1bce6a9187a52a240e648e19d09bac733e5194743d01d27daf14ad0 not found: ID does not exist"
Nov 03 10:50:01.491431 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.491353    5393 scope.go:122] "RemoveContainer" containerID="ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e"
Nov 03 10:50:01.496984 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:01.491873    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e\": container with ID starting with ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e not found: ID does not exist" containerID="ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e"
Nov 03 10:50:01.497139 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:01.491900    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e"} err="failed to get container status \"ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e\": rpc error: code = NotFound desc = could not find container \"ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e\": container with ID starting with ddafd75c159a856f7849a4a17288cc221d88ee3dea7ba6b1da517cf7779e0b1e not found: ID does not exist"
Nov 03 10:50:02.252178 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:02.250124    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" path="/var/lib/kubelet/pods/28d4380b-8e6e-4505-a07e-4bf980ec7296/volumes"
Nov 03 10:50:02.252178 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:02.250812    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" path="/var/lib/kubelet/pods/d2b40ca1-4b45-4ab2-8c80-5c7b4215d778/volumes"
Nov 03 10:50:03.937069 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:03.937023    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:03.942343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:03.938056    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:03.955944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:03.948977    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.004945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:03.989110    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.046683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.046093    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tq59l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:50:04.046683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.046169    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.082079 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.082024    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.250815 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.250719    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:50:04.328088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.328041    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.787957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.785012    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:50:04.938946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.938008    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.938946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.938780    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:50:04.950955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.947991    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:50:04.950955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.948640    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:50:04.950955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.949499    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:04.950955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:04.949796    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:50:05.006129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.003018    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:05.006129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.003728    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:50:05.332010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.331964    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 10:50:05.333544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.333519    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:50:05.418407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.418295    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:05.627075 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.625534    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:50:05.637046 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.637017    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:50:05.881940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.881827    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:50:05.940209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.940132    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:05.970148 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:05.960137    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:06.956085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:06.956032    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:07.022579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:07.022516    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:07.129097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:07.129053    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d7gsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:50:07.129377 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:07.129330    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:07.246996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:07.245593    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:50:07.265959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:07.263994    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:07.375944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:07.375687    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:50:08.047940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:08.041523    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:50:09.086946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:09.073419    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"]
Nov 03 10:50:09.106707 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:09.106616    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7d2989-glbdx"]
Nov 03 10:50:10.232675 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:10.232592    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bed52ed-e881-4545-8aac-5c17bcee11ef" path="/var/lib/kubelet/pods/1bed52ed-e881-4545-8aac-5c17bcee11ef/volumes"
Nov 03 10:50:13.191849 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:13.191804    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:50:13.440529 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:13.440494    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:50:13.440730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:13.440560    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:50:14.025312 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.025267    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"]
Nov 03 10:50:14.028930 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.028884    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c188f69f-qxl8r"]
Nov 03 10:50:14.233681 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.233625    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f56a720-bdac-485d-ae61-f983f408a10a" path="/var/lib/kubelet/pods/6f56a720-bdac-485d-ae61-f983f408a10a/volumes"
Nov 03 10:50:14.338000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.337241    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": read tcp 10.134.0.2:50418->10.134.0.101:15021: read: connection reset by peer" start-of-body=
Nov 03 10:50:14.338000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.337307    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": read tcp 10.134.0.2:50418->10.134.0.101:15021: read: connection reset by peer"
Nov 03 10:50:14.856389 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.856349    5393 generic.go:356] "Generic (PLEG): container finished" podID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerID="210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da" exitCode=0
Nov 03 10:50:14.856626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.856413    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" event={"ID":"d8395d93-9b56-4476-a706-3e00b3847ef6","Type":"ContainerDied","Data":"210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da"}
Nov 03 10:50:14.856933 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.856899    5393 scope.go:122] "RemoveContainer" containerID="210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da"
Nov 03 10:50:14.858280 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.858255    5393 generic.go:356] "Generic (PLEG): container finished" podID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerID="8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1" exitCode=0
Nov 03 10:50:14.858391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.858373    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" event={"ID":"f78765ce-ebc8-4d04-8ae6-721dddd8ba72","Type":"ContainerDied","Data":"8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1"}
Nov 03 10:50:14.858635 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.858607    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:14.858776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.858687    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:14.858776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.858739    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:14.858776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.858749    5393 scope.go:122] "RemoveContainer" containerID="8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1"
Nov 03 10:50:14.860213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.860188    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:14.860292 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.860245    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:14.860292 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:14.860276    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:15.959344 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:15.959305    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" event={"ID":"f78765ce-ebc8-4d04-8ae6-721dddd8ba72","Type":"ContainerStarted","Data":"496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183"}
Nov 03 10:50:15.969432 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:15.969144    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" event={"ID":"d8395d93-9b56-4476-a706-3e00b3847ef6","Type":"ContainerStarted","Data":"9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0"}
Nov 03 10:50:16.566690 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:16.566654    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:50:16.566893 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:16.566870    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:50:16.566965 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:16.566950    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 10:50:16.915945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:16.913309    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:50:16.915945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:16.913433    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 10:50:16.915945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:16.913514    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 10:50:17.567153 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:17.567071    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:50:17.567153 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:17.567127    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 10:50:17.915946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:17.915370    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 10:50:17.915946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:17.915457    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 10:50:18.570954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:18.567325    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:50:18.570954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:18.567388    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 10:50:18.912944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:18.912313    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 10:50:18.912944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:18.912375    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 10:50:19.576945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:19.569308    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:50:19.576945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:19.569385    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 10:50:19.916943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:19.916306    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 10:50:19.916943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:19.916369    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 10:50:20.572957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.572799    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:50:20.572957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.572859    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 10:50:20.831464 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.831363    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"]
Nov 03 10:50:20.846946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.832819    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" containerName="payload-processing"
Nov 03 10:50:20.869011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.868977    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" containerName="payload-processing"
Nov 03 10:50:20.869305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.869285    5393 cpu_manager.go:414] "containerMap: removing container" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy"
Nov 03 10:50:20.869421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.869408    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy"
Nov 03 10:50:20.869562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.869549    5393 cpu_manager.go:414] "containerMap: removing container" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy"
Nov 03 10:50:20.869652 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.869640    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy"
Nov 03 10:50:20.869999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.869980    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy"
Nov 03 10:50:20.870133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.870121    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b40ca1-4b45-4ab2-8c80-5c7b4215d778" containerName="payload-processing"
Nov 03 10:50:20.870242 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.870231    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d4380b-8e6e-4505-a07e-4bf980ec7296" containerName="istio-proxy"
Nov 03 10:50:20.874402 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.874374    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"]
Nov 03 10:50:20.874513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.874489    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.878822 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.878799    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-814dc0-openshift-default-dockercfg-wx4zt\""
Nov 03 10:50:20.936114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.936074    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 10:50:20.936414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.936368    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.973860    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istiod-ca-cert\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.973997    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-socket\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.974093    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-credential-socket\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.974122    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-token\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.974161    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfcm7\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-kube-api-access-pfcm7\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.974256    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-data\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.974295    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-envoy\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.974372    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-certs\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:20.982961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:20.974432    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-podinfo\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.135945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.110851    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istiod-ca-cert\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.136347 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.136305    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-socket\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.177159 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.177121    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-credential-socket\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.177898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.177872    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-token\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.178117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.178097    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcm7\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-kube-api-access-pfcm7\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.209984 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.142998    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istiod-ca-cert\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.237968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.143314    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-socket\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.237968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.216393    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-token\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.237968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.177771    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-credential-socket\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.251962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.238638    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-data\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.251962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.238719    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-envoy\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.251962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.238801    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-certs\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.251962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.238861    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-podinfo\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.274968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.255720    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-data\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.274968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.256370    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-certs\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.286937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.286884    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfcm7\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-kube-api-access-pfcm7\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.330058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.330022    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-envoy\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.334506 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.334471    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-podinfo\") pod \"e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") " pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.573471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.573378    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 10:50:21.573751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.573721    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 10:50:21.620945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.620271    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:21.911201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.911164    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 03 10:50:21.911635 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:21.911233    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 03 10:50:22.004050 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.003983    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"]
Nov 03 10:50:22.064105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.062776    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:22.077778 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.074815    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"]
Nov 03 10:50:22.078106 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.078073    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:22.078329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.078302    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:22.127948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.126491    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96b3cb37-n4b8z"]
Nov 03 10:50:22.198556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.198516    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:50:22.230894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.230865    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8da127-97af-43f8-b6b2-3dd18a12b072" path="/var/lib/kubelet/pods/ac8da127-97af-43f8-b6b2-3dd18a12b072/volumes"
Nov 03 10:50:22.231211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.231193    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" event={"ID":"a8c0b587-9ed2-4cbc-922a-74782922dc1a","Type":"ContainerStarted","Data":"a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263"}
Nov 03 10:50:22.231270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.231243    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" event={"ID":"a8c0b587-9ed2-4cbc-922a-74782922dc1a","Type":"ContainerStarted","Data":"25daac67120701a534e2fbbbf4a1bf2fb9809285d511edac82c7241dd2116633"}
Nov 03 10:50:22.243672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.243611    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podStartSLOduration=2.24359293 podStartE2EDuration="2.24359293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:22.241147807 +0000 UTC m=+1266.146002249" watchObservedRunningTime="2026-11-03 10:50:22.24359293 +0000 UTC m=+1266.148447372"
Nov 03 10:50:22.571980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.571882    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:50:22.573156 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.573133    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:50:22.622820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.622784    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:22.932382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.932350    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:50:22.933253 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.933225    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:50:22.934351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:22.934334    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:50:23.663778 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:23.663073    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:24.296250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.296223    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:24.304204 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.301378    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:50:24.890939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.888815    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"]
Nov 03 10:50:24.911945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.908854    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:24.912618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.912585    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"]
Nov 03 10:50:24.913699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.913677    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a51d55\"/\"default-dockercfg-hl8qb\""
Nov 03 10:50:24.915011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.914977    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a51d55\"/\"ipp-route-a51d55-kserve-self-signed-certs\""
Nov 03 10:50:24.916569 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.916549    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a51d55\"/\"openshift-service-ca.crt\""
Nov 03 10:50:24.916754 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:24.916740    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a51d55\"/\"kube-root-ca.crt\""
Nov 03 10:50:25.007605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.007576    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-home\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.007783 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.007652    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-dshm\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.007856 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.007830    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tls-certs\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.007981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.007958    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnws\" (UniqueName: \"kubernetes.io/projected/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-kube-api-access-kdnws\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.008134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.008116    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tmp-dir\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.008188 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.008160    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-model-cache\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.128708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.127762    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tls-certs\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.128708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.127870    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnws\" (UniqueName: \"kubernetes.io/projected/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-kube-api-access-kdnws\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.128708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.128011    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tmp-dir\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.128708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.128060    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-model-cache\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.128708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.128094    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-home\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.128708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.128187    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-dshm\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.143004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.130313    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tmp-dir\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.143004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.132647    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-model-cache\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.143004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.133021    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-home\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.143004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.140597    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tls-certs\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.152966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.147316    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnws\" (UniqueName: \"kubernetes.io/projected/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-kube-api-access-kdnws\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.152966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.151194    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-dshm\") pod \"ipp-route-a51d55-kserve-7745757bbf-nlgvc\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") " pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:25.238954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:25.237435    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:26.146013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.124971    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"]
Nov 03 10:50:26.222993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.209712    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"]
Nov 03 10:50:26.661570 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.661542    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:26.745667 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.745637    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-814dc0-dockercfg-shbrg\""
Nov 03 10:50:26.745845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.745731    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-814dc0\""
Nov 03 10:50:26.756215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.751391    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc" event={"ID":"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee","Type":"ContainerStarted","Data":"c8f058ffffcf9796005163967fd6daaf9506b1168ee64ba72f343d02960c583f"}
Nov 03 10:50:26.756215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.751434    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"]
Nov 03 10:50:26.756215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.753101    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"]
Nov 03 10:50:26.757329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.757300    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7mb\" (UniqueName: \"kubernetes.io/projected/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-kube-api-access-9g7mb\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj\" (UID: \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:26.757620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.757600    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj\" (UID: \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:26.761048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.761027    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"]
Nov 03 10:50:26.761195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.761127    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:26.858966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.858929    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7mb\" (UniqueName: \"kubernetes.io/projected/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-kube-api-access-9g7mb\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj\" (UID: \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:26.859135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.859119    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj\" (UID: \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:26.859894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.859873    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj\" (UID: \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:26.869865 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.869838    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7mb\" (UniqueName: \"kubernetes.io/projected/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-kube-api-access-9g7mb\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj\" (UID: \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:26.973779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.971955    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf858d52-3486-4db1-9a50-243e0ee683a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl\" (UID: \"bf858d52-3486-4db1-9a50-243e0ee683a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:26.973779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:26.973069    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whj4\" (UniqueName: \"kubernetes.io/projected/bf858d52-3486-4db1-9a50-243e0ee683a5-kube-api-access-4whj4\") pod \"payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl\" (UID: \"bf858d52-3486-4db1-9a50-243e0ee683a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:27.074882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:27.074552    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whj4\" (UniqueName: \"kubernetes.io/projected/bf858d52-3486-4db1-9a50-243e0ee683a5-kube-api-access-4whj4\") pod \"payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl\" (UID: \"bf858d52-3486-4db1-9a50-243e0ee683a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:27.074882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:27.074650    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf858d52-3486-4db1-9a50-243e0ee683a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl\" (UID: \"bf858d52-3486-4db1-9a50-243e0ee683a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:27.075675 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:27.075653    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf858d52-3486-4db1-9a50-243e0ee683a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl\" (UID: \"bf858d52-3486-4db1-9a50-243e0ee683a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:27.103695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:27.103662    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whj4\" (UniqueName: \"kubernetes.io/projected/bf858d52-3486-4db1-9a50-243e0ee683a5-kube-api-access-4whj4\") pod \"payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl\" (UID: \"bf858d52-3486-4db1-9a50-243e0ee683a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:27.134203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:27.134163    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:27.146486 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:27.145492    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:28.176817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:28.176781    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc" event={"ID":"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee","Type":"ContainerStarted","Data":"61ae8692d82963cacae28678c5bc0d6ccf36695cb5f5c8b6d45e125c9339f482"}
Nov 03 10:50:28.283943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:28.282442    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc" podStartSLOduration=4.282423262 podStartE2EDuration="4.282423262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:28.213631538 +0000 UTC m=+1272.118485980" watchObservedRunningTime="2026-11-03 10:50:28.282423262 +0000 UTC m=+1272.187277704"
Nov 03 10:50:28.592003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:28.586513    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:28.676950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:28.676461    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:28.679360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:28.679331    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 10:50:28.766487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:28.766436    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:50:28.833498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:28.833146    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"]
Nov 03 10:50:28.927053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:28.927012    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:29.064521 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:50:29.064482    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7aa82b_84d0_4e85_a2d9_e0e9541d6358.slice/crio-511e96c80a30b7448b9da9bb6090e57023119fc863744510aaf2d3043e7a68c4 WatchSource:0}: Error finding container 511e96c80a30b7448b9da9bb6090e57023119fc863744510aaf2d3043e7a68c4: Status 404 returned error can't find the container with id 511e96c80a30b7448b9da9bb6090e57023119fc863744510aaf2d3043e7a68c4
Nov 03 10:50:29.282423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.282379    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:29.307487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.307445    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl" event={"ID":"bf858d52-3486-4db1-9a50-243e0ee683a5","Type":"ContainerStarted","Data":"f109aa1384ae6025a9d5017cda159284acbeed7aa2763d0f1c131d211c56030e"}
Nov 03 10:50:29.352168 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.352129    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj" event={"ID":"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358","Type":"ContainerStarted","Data":"511e96c80a30b7448b9da9bb6090e57023119fc863744510aaf2d3043e7a68c4"}
Nov 03 10:50:29.445417 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.444527    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"]
Nov 03 10:50:29.567956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.559966    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"]
Nov 03 10:50:29.678063 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.677967    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"]
Nov 03 10:50:29.742256 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.741987    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"]
Nov 03 10:50:29.803848 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.803816    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"]
Nov 03 10:50:29.895497 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.877250    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f575da8-hx6lh"]
Nov 03 10:50:29.955054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.939783    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:50:29.955054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.940773    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:50:29.977734 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:29.977702    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d476da-4csgp"]
Nov 03 10:50:30.022862 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:30.022830    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:50:30.330392 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:30.330333    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c362709-1ebc-4d12-83ae-655af32fb700" path="/var/lib/kubelet/pods/1c362709-1ebc-4d12-83ae-655af32fb700/volumes"
Nov 03 10:50:30.331847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:30.331821    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72fc1a45-3ec7-49a7-ba99-6bcf03ae4341" path="/var/lib/kubelet/pods/72fc1a45-3ec7-49a7-ba99-6bcf03ae4341/volumes"
Nov 03 10:50:30.362540 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:30.362496    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl" event={"ID":"bf858d52-3486-4db1-9a50-243e0ee683a5","Type":"ContainerStarted","Data":"42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef"}
Nov 03 10:50:30.362712 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:30.362581    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:30.364777 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:30.364745    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj" event={"ID":"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358","Type":"ContainerStarted","Data":"95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989"}
Nov 03 10:50:30.402945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:30.399204    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl" podStartSLOduration=4.399181166 podStartE2EDuration="4.399181166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:30.379911414 +0000 UTC m=+1274.284765849" watchObservedRunningTime="2026-11-03 10:50:30.399181166 +0000 UTC m=+1274.304035609"
Nov 03 10:50:30.423656 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:30.423588    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj" podStartSLOduration=4.423568459 podStartE2EDuration="4.423568459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:30.419355676 +0000 UTC m=+1274.324210141" watchObservedRunningTime="2026-11-03 10:50:30.423568459 +0000 UTC m=+1274.328422901"
Nov 03 10:50:31.021099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:31.021065    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"]
Nov 03 10:50:31.024524 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:31.024492    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9b26426-ctz7l"]
Nov 03 10:50:31.371630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:31.371597    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:32.259257 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:32.259220    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3" path="/var/lib/kubelet/pods/1f4af4af-8ed8-448a-96ae-1ae14fe8f4b3/volumes"
Nov 03 10:50:32.927996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:32.927808    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"]
Nov 03 10:50:32.934727 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:32.934656    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-38c9b0-9r8z5"]
Nov 03 10:50:33.833893 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:33.833848    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5" podUID="54f18330-9312-499f-96d7-37f6c77dd1a1" containerName="payload-processing" containerID="cri-o://05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175" gracePeriod=25
Nov 03 10:50:34.176857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.176183    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:50:34.252305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.251246    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f18330-9312-499f-96d7-37f6c77dd1a1-kube-api-access-2mvvk\" (UniqueName: \"kubernetes.io/projected/54f18330-9312-499f-96d7-37f6c77dd1a1-kube-api-access-2mvvk\") pod \"54f18330-9312-499f-96d7-37f6c77dd1a1\" (UID: \"54f18330-9312-499f-96d7-37f6c77dd1a1\") "
Nov 03 10:50:34.267549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.267508    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f18330-9312-499f-96d7-37f6c77dd1a1-kube-api-access-2mvvk" pod "54f18330-9312-499f-96d7-37f6c77dd1a1" (UID: "54f18330-9312-499f-96d7-37f6c77dd1a1"). InnerVolumeSpecName "kube-api-access-2mvvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:34.277528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.277485    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4388dcef-532e-4854-9846-1a5701465758" path="/var/lib/kubelet/pods/4388dcef-532e-4854-9846-1a5701465758/volumes"
Nov 03 10:50:34.352695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.352661    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f18330-9312-499f-96d7-37f6c77dd1a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f18330-9312-499f-96d7-37f6c77dd1a1-plugins-config-volume\") pod \"54f18330-9312-499f-96d7-37f6c77dd1a1\" (UID: \"54f18330-9312-499f-96d7-37f6c77dd1a1\") "
Nov 03 10:50:34.353113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.353085    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f18330-9312-499f-96d7-37f6c77dd1a1-plugins-config-volume" pod "54f18330-9312-499f-96d7-37f6c77dd1a1" (UID: "54f18330-9312-499f-96d7-37f6c77dd1a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:50:34.353427 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.353412    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f18330-9312-499f-96d7-37f6c77dd1a1-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:34.353468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.353432    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mvvk\" (UniqueName: \"kubernetes.io/projected/54f18330-9312-499f-96d7-37f6c77dd1a1-kube-api-access-2mvvk\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:34.405821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.405729    5393 generic.go:356] "Generic (PLEG): container finished" podID="54f18330-9312-499f-96d7-37f6c77dd1a1" containerID="05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175" exitCode=0
Nov 03 10:50:34.405821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.405777    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5" event={"ID":"54f18330-9312-499f-96d7-37f6c77dd1a1","Type":"ContainerDied","Data":"05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175"}
Nov 03 10:50:34.406113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.405829    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5" event={"ID":"54f18330-9312-499f-96d7-37f6c77dd1a1","Type":"ContainerDied","Data":"ca6ba131eec8bd27cedc1a9a9e271c2ca840065e97633b4fc205f895ef802f7b"}
Nov 03 10:50:34.406113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.405781    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"
Nov 03 10:50:34.406113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.405866    5393 scope.go:122] "RemoveContainer" containerID="05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175"
Nov 03 10:50:34.417654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.417633    5393 scope.go:122] "RemoveContainer" containerID="05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175"
Nov 03 10:50:34.418134 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:34.418101    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175\": container with ID starting with 05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175 not found: ID does not exist" containerID="05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175"
Nov 03 10:50:34.418276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.418142    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175"} err="failed to get container status \"05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175\": rpc error: code = NotFound desc = could not find container \"05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175\": container with ID starting with 05633c567a1ec63be7c9d7164584ad9f87b16791931c6f2682ff0d04ccae3175 not found: ID does not exist"
Nov 03 10:50:34.431804 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.431769    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"]
Nov 03 10:50:34.440501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.440468    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a2697c-f897bd975-shvn5"]
Nov 03 10:50:34.445664 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.445623    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd" podUID="ded51902-10c7-474c-bb34-a8b97bc86562" containerName="payload-pre-processing" containerID="cri-o://20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8" gracePeriod=25
Nov 03 10:50:34.726214 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.726192    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:50:34.761088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.761048    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded51902-10c7-474c-bb34-a8b97bc86562-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded51902-10c7-474c-bb34-a8b97bc86562-plugins-config-volume\") pod \"ded51902-10c7-474c-bb34-a8b97bc86562\" (UID: \"ded51902-10c7-474c-bb34-a8b97bc86562\") "
Nov 03 10:50:34.761306 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.761283    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded51902-10c7-474c-bb34-a8b97bc86562-kube-api-access-z6f9f\" (UniqueName: \"kubernetes.io/projected/ded51902-10c7-474c-bb34-a8b97bc86562-kube-api-access-z6f9f\") pod \"ded51902-10c7-474c-bb34-a8b97bc86562\" (UID: \"ded51902-10c7-474c-bb34-a8b97bc86562\") "
Nov 03 10:50:34.761668 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.761589    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded51902-10c7-474c-bb34-a8b97bc86562-plugins-config-volume" pod "ded51902-10c7-474c-bb34-a8b97bc86562" (UID: "ded51902-10c7-474c-bb34-a8b97bc86562"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:50:34.762053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.762023    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded51902-10c7-474c-bb34-a8b97bc86562-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:34.764598 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.764557    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded51902-10c7-474c-bb34-a8b97bc86562-kube-api-access-z6f9f" pod "ded51902-10c7-474c-bb34-a8b97bc86562" (UID: "ded51902-10c7-474c-bb34-a8b97bc86562"). InnerVolumeSpecName "kube-api-access-z6f9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:34.863179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:34.863144    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6f9f\" (UniqueName: \"kubernetes.io/projected/ded51902-10c7-474c-bb34-a8b97bc86562-kube-api-access-z6f9f\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:35.238874 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.238844    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:35.257105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.257059    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:35.418237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.418205    5393 generic.go:356] "Generic (PLEG): container finished" podID="ded51902-10c7-474c-bb34-a8b97bc86562" containerID="20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8" exitCode=0
Nov 03 10:50:35.418441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.418270    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"
Nov 03 10:50:35.418441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.418405    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd" event={"ID":"ded51902-10c7-474c-bb34-a8b97bc86562","Type":"ContainerDied","Data":"20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8"}
Nov 03 10:50:35.418560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.418442    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd" event={"ID":"ded51902-10c7-474c-bb34-a8b97bc86562","Type":"ContainerDied","Data":"d81ded4935e45ae4fbc2fc5f1985a4da3a20b47f07d71388f6814356da54224a"}
Nov 03 10:50:35.418560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.418461    5393 scope.go:122] "RemoveContainer" containerID="20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8"
Nov 03 10:50:35.433604 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.433582    5393 scope.go:122] "RemoveContainer" containerID="20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8"
Nov 03 10:50:35.434011 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:35.433985    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8\": container with ID starting with 20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8 not found: ID does not exist" containerID="20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8"
Nov 03 10:50:35.434142 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.434020    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8"} err="failed to get container status \"20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8\": rpc error: code = NotFound desc = could not find container \"20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8\": container with ID starting with 20d13cb5d5cd8158fce031aa6ab4a38e6af73cf52fc1ae81266d876e76e547d8 not found: ID does not exist"
Nov 03 10:50:35.437713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.437690    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:50:35.443761 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.443735    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"]
Nov 03 10:50:35.451325 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:35.451292    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-6f7df5f85f-wh5qd"]
Nov 03 10:50:36.239179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.239147    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f18330-9312-499f-96d7-37f6c77dd1a1" path="/var/lib/kubelet/pods/54f18330-9312-499f-96d7-37f6c77dd1a1/volumes"
Nov 03 10:50:36.239563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.239548    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded51902-10c7-474c-bb34-a8b97bc86562" path="/var/lib/kubelet/pods/ded51902-10c7-474c-bb34-a8b97bc86562/volumes"
Nov 03 10:50:36.246065 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.246038    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"]
Nov 03 10:50:36.246595 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.246581    5393 cpu_manager.go:414] "containerMap: removing container" podUID="54f18330-9312-499f-96d7-37f6c77dd1a1" containerName="payload-processing"
Nov 03 10:50:36.246661 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.246597    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f18330-9312-499f-96d7-37f6c77dd1a1" containerName="payload-processing"
Nov 03 10:50:36.246661 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.246629    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ded51902-10c7-474c-bb34-a8b97bc86562" containerName="payload-pre-processing"
Nov 03 10:50:36.246661 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.246634    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded51902-10c7-474c-bb34-a8b97bc86562" containerName="payload-pre-processing"
Nov 03 10:50:36.246770 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.246724    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f18330-9312-499f-96d7-37f6c77dd1a1" containerName="payload-processing"
Nov 03 10:50:36.246770 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.246738    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded51902-10c7-474c-bb34-a8b97bc86562" containerName="payload-pre-processing"
Nov 03 10:50:36.251216 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.251197    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:36.257038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.257001    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a2697c\""
Nov 03 10:50:36.261102 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.260847    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"]
Nov 03 10:50:36.289952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.287380    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a69b56-ebe9-4c52-aed2-aebc05bc4344-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t\" (UID: \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:36.289952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.287433    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4qjq\" (UniqueName: \"kubernetes.io/projected/62a69b56-ebe9-4c52-aed2-aebc05bc4344-kube-api-access-s4qjq\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t\" (UID: \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:36.395172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.395134    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a69b56-ebe9-4c52-aed2-aebc05bc4344-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t\" (UID: \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:36.406945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.395485    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qjq\" (UniqueName: \"kubernetes.io/projected/62a69b56-ebe9-4c52-aed2-aebc05bc4344-kube-api-access-s4qjq\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t\" (UID: \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:36.430944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.408737    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a69b56-ebe9-4c52-aed2-aebc05bc4344-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t\" (UID: \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:36.444980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.441652    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qjq\" (UniqueName: \"kubernetes.io/projected/62a69b56-ebe9-4c52-aed2-aebc05bc4344-kube-api-access-s4qjq\") pod \"payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t\" (UID: \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:36.573022 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.571493    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:36.997243 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:36.995747    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"]
Nov 03 10:50:37.022943 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:50:37.014055    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a69b56_ebe9_4c52_aed2_aebc05bc4344.slice/crio-884a959dce5b124c6fde3fae8be8c2bc941c2037e0a16364d24caa44c7f06792 WatchSource:0}: Error finding container 884a959dce5b124c6fde3fae8be8c2bc941c2037e0a16364d24caa44c7f06792: Status 404 returned error can't find the container with id 884a959dce5b124c6fde3fae8be8c2bc941c2037e0a16364d24caa44c7f06792
Nov 03 10:50:37.517709 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:37.517663    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t" event={"ID":"62a69b56-ebe9-4c52-aed2-aebc05bc4344","Type":"ContainerStarted","Data":"b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde"}
Nov 03 10:50:37.518261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:37.517790    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t" event={"ID":"62a69b56-ebe9-4c52-aed2-aebc05bc4344","Type":"ContainerStarted","Data":"884a959dce5b124c6fde3fae8be8c2bc941c2037e0a16364d24caa44c7f06792"}
Nov 03 10:50:37.518413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:37.518392    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:37.557013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:37.541704    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t" podStartSLOduration=1.541683978 podStartE2EDuration="1.541683978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:37.536190512 +0000 UTC m=+1281.441044953" watchObservedRunningTime="2026-11-03 10:50:37.541683978 +0000 UTC m=+1281.446538420"
Nov 03 10:50:38.031585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.031553    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"]
Nov 03 10:50:38.036429 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.036406    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607e9ee7-vbsbf"]
Nov 03 10:50:38.404643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.404308    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d39b59-b897-49c4-8b72-6238db5244b7" path="/var/lib/kubelet/pods/17d39b59-b897-49c4-8b72-6238db5244b7/volumes"
Nov 03 10:50:38.559287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.559248    5393 generic.go:356] "Generic (PLEG): container finished" podID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerID="287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0" exitCode=0
Nov 03 10:50:38.559880 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.559846    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" event={"ID":"ce655ad0-1a5a-4615-863a-9fd43685b4bb","Type":"ContainerDied","Data":"287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0"}
Nov 03 10:50:38.560350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.560330    5393 scope.go:122] "RemoveContainer" containerID="287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0"
Nov 03 10:50:38.562377 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.562146    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:38.562785 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.562744    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:38.562906 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:38.562891    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:39.579699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:39.579667    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" event={"ID":"ce655ad0-1a5a-4615-863a-9fd43685b4bb","Type":"ContainerStarted","Data":"04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a"}
Nov 03 10:50:39.619369 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:39.619344    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:50:39.620406 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:39.620375    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 10:50:39.620617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:39.620601    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 10:50:40.136993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:40.131852    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"]
Nov 03 10:50:40.136993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:40.132149    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy" containerID="cri-o://d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a" gracePeriod=30
Nov 03 10:50:40.629243 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:40.629204    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 10:50:40.629774 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:40.629273    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 10:50:41.375505 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:41.375474    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:41.619453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:41.619419    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 10:50:41.619626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:41.619485    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 10:50:42.015038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:42.015006    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:50:42.408086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:42.408056    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:42.627949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:42.627735    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 10:50:42.627949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:42.627806    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 10:50:43.214527 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:43.214494    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:50:43.619516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:43.619474    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 10:50:43.619693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:43.619539    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 10:50:44.618770 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:44.618732    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 10:50:44.619211 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:44.618801    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 10:50:45.528066 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.528039    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:50:45.624385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.624356    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:50:45.625702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.625683    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:50:45.630189 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630165    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-credential-socket\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.630322 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630205    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-socket\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.630322 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630246    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-kube-api-access-m9wnj\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-kube-api-access-m9wnj\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.630322 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630283    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-certs\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.630487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630434    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-podinfo\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.630487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630449    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-credential-socket" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:45.630587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630553    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-data\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.630641 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630582    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-certs" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:45.630641 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630595    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-socket" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:45.630741 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630654    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-envoy\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.630793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630776    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-data" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:45.630852 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630802    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istiod-ca-cert\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.630852 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.630830    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-token\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-token\") pod \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\" (UID: \"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0\") "
Nov 03 10:50:45.631164 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.631144    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istiod-ca-cert" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:50:45.631436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.631418    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.631492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.631437    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.631492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.631447    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.631492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.631476    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.631492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.631484    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.633410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.633384    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-podinfo" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:50:45.633529 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.633440    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-kube-api-access-m9wnj" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "kube-api-access-m9wnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:45.633636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.633613    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-envoy" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:50:45.633699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.633628    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-token" pod "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" (UID: "f04b7efd-1fdb-4ac2-b9bf-13765816ddd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:45.732713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.732675    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.732898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.732803    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.732898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.732836    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9wnj\" (UniqueName: \"kubernetes.io/projected/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-kube-api-access-m9wnj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.732898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.732856    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:45.745032 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.745001    5393 generic.go:356] "Generic (PLEG): container finished" podID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerID="d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a" exitCode=0
Nov 03 10:50:45.745191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.745050    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"
Nov 03 10:50:45.745191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.745079    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" event={"ID":"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0","Type":"ContainerDied","Data":"d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a"}
Nov 03 10:50:45.745191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.745143    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2" event={"ID":"f04b7efd-1fdb-4ac2-b9bf-13765816ddd0","Type":"ContainerDied","Data":"7e675b8d8e540e904d33c846912ec98863cb9615ef9be789c071a60201e9564e"}
Nov 03 10:50:45.745191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.745170    5393 scope.go:122] "RemoveContainer" containerID="d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a"
Nov 03 10:50:45.756769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.756753    5393 scope.go:122] "RemoveContainer" containerID="09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff"
Nov 03 10:50:45.786890 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.784429    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"]
Nov 03 10:50:45.787104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.787071    5393 scope.go:122] "RemoveContainer" containerID="d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a"
Nov 03 10:50:45.787551 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:45.787522    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a\": container with ID starting with d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a not found: ID does not exist" containerID="d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a"
Nov 03 10:50:45.787681 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.787560    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a"} err="failed to get container status \"d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a\": rpc error: code = NotFound desc = could not find container \"d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a\": container with ID starting with d0b1685def8697763a563cfbbecd48c25126562e3c55a499e19371080595c04a not found: ID does not exist"
Nov 03 10:50:45.787681 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.787586    5393 scope.go:122] "RemoveContainer" containerID="09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff"
Nov 03 10:50:45.787965 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:45.787898    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff\": container with ID starting with 09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff not found: ID does not exist" containerID="09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff"
Nov 03 10:50:45.788045 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.787978    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff"} err="failed to get container status \"09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff\": rpc error: code = NotFound desc = could not find container \"09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff\": container with ID starting with 09db393afb7c1e0fd0cb175399506c58c985506d21653060e2debdd5fc13d7ff not found: ID does not exist"
Nov 03 10:50:45.791370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:45.791349    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-38c9b0-openshift-default-778969b477-d74m2"]
Nov 03 10:50:46.231447 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:46.231413    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" path="/var/lib/kubelet/pods/f04b7efd-1fdb-4ac2-b9bf-13765816ddd0/volumes"
Nov 03 10:50:48.563495 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:48.563464    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:50:49.387320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:49.387285    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"]
Nov 03 10:50:49.398821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:49.398778    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"]
Nov 03 10:50:51.031038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:51.030998    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"]
Nov 03 10:50:51.034971 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:51.034948    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51f25692-x9dzz"]
Nov 03 10:50:52.231304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.231276    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fb66bc-0987-4331-acfe-861e694099a8" path="/var/lib/kubelet/pods/a8fb66bc-0987-4331-acfe-861e694099a8/volumes"
Nov 03 10:50:52.552847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.552762    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"]
Nov 03 10:50:52.553728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.553702    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy"
Nov 03 10:50:52.553851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.553731    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy"
Nov 03 10:50:52.553851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.553758    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy"
Nov 03 10:50:52.553851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.553766    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy"
Nov 03 10:50:52.554053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.554037    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy"
Nov 03 10:50:52.554107 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.554059    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04b7efd-1fdb-4ac2-b9bf-13765816ddd0" containerName="istio-proxy"
Nov 03 10:50:52.559763 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.559743    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.567038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.567013    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b3dc04-openshift-default-dockercfg-4cq8t\""
Nov 03 10:50:52.571748 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.571720    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"]
Nov 03 10:50:52.698943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.662988    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea688a05-d023-4385-8f9c-54ceac47a320-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.698943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.663075    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.698943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.663159    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.698943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.663194    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.698943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.663227    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.698943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.663393    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.698943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.663426    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.698943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.663470    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea688a05-d023-4385-8f9c-54ceac47a320-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.699610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.699094    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qm7\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-kube-api-access-r7qm7\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.817224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.817134    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.817796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.817528    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.818036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.818008    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.818391 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.818363    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.818829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.818811    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.819010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.818986    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea688a05-d023-4385-8f9c-54ceac47a320-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.819601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.819155    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qm7\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-kube-api-access-r7qm7\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.819852 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.819835    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea688a05-d023-4385-8f9c-54ceac47a320-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.820328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.820067    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.842984 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.836716    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.928014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.927979    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.928514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.928480    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.928872 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.928845    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.967123 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.948945    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea688a05-d023-4385-8f9c-54ceac47a320-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.998970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.982611    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:52.999237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:52.993390    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:53.070458 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:53.070374    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qm7\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-kube-api-access-r7qm7\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:53.071253 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:53.071214    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea688a05-d023-4385-8f9c-54ceac47a320-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:53.178119 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:53.178090    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:53.524531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:53.524484    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"]
Nov 03 10:50:53.543351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:53.543307    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:53.543506 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:53.543399    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:53.543506 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:53.543473    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:50:54.009847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.009805    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" event={"ID":"ea688a05-d023-4385-8f9c-54ceac47a320","Type":"ContainerStarted","Data":"92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8"}
Nov 03 10:50:54.010045 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.009877    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" event={"ID":"ea688a05-d023-4385-8f9c-54ceac47a320","Type":"ContainerStarted","Data":"f81532d6c3f955f79dca3a31d02241371685caf5798ae58c142e315c895b5bcd"}
Nov 03 10:50:54.028875 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.028822    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podStartSLOduration=2.02880825 podStartE2EDuration="2.02880825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:54.028084619 +0000 UTC m=+1297.932939073" watchObservedRunningTime="2026-11-03 10:50:54.02880825 +0000 UTC m=+1297.933662690"
Nov 03 10:50:54.181487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.181456    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:54.214330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.214300    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:54.387932 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.387887    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl" podUID="bf858d52-3486-4db1-9a50-243e0ee683a5" containerName="payload-processing" containerID="cri-o://42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef" gracePeriod=25
Nov 03 10:50:54.399386 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.399349    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj" podUID="1a7aa82b-84d0-4e85-a2d9-e0e9541d6358" containerName="payload-pre-processing" containerID="cri-o://95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989" gracePeriod=25
Nov 03 10:50:54.703272 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.703247    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:54.729846 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.729821    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:54.831412 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.831373    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-kube-api-access-9g7mb\" (UniqueName: \"kubernetes.io/projected/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-kube-api-access-9g7mb\") pod \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\" (UID: \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\") "
Nov 03 10:50:54.831644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.831627    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf858d52-3486-4db1-9a50-243e0ee683a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf858d52-3486-4db1-9a50-243e0ee683a5-plugins-config-volume\") pod \"bf858d52-3486-4db1-9a50-243e0ee683a5\" (UID: \"bf858d52-3486-4db1-9a50-243e0ee683a5\") "
Nov 03 10:50:54.831711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.831695    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf858d52-3486-4db1-9a50-243e0ee683a5-kube-api-access-4whj4\" (UniqueName: \"kubernetes.io/projected/bf858d52-3486-4db1-9a50-243e0ee683a5-kube-api-access-4whj4\") pod \"bf858d52-3486-4db1-9a50-243e0ee683a5\" (UID: \"bf858d52-3486-4db1-9a50-243e0ee683a5\") "
Nov 03 10:50:54.832067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.831749    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-plugins-config-volume\") pod \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\" (UID: \"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358\") "
Nov 03 10:50:54.832190 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.832107    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf858d52-3486-4db1-9a50-243e0ee683a5-plugins-config-volume" pod "bf858d52-3486-4db1-9a50-243e0ee683a5" (UID: "bf858d52-3486-4db1-9a50-243e0ee683a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:50:54.832324 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.832178    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-plugins-config-volume" pod "1a7aa82b-84d0-4e85-a2d9-e0e9541d6358" (UID: "1a7aa82b-84d0-4e85-a2d9-e0e9541d6358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:50:54.832588 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.832568    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:54.832634 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.832597    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf858d52-3486-4db1-9a50-243e0ee683a5-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:54.833866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.833844    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-kube-api-access-9g7mb" pod "1a7aa82b-84d0-4e85-a2d9-e0e9541d6358" (UID: "1a7aa82b-84d0-4e85-a2d9-e0e9541d6358"). InnerVolumeSpecName "kube-api-access-9g7mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:54.833955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.833866    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf858d52-3486-4db1-9a50-243e0ee683a5-kube-api-access-4whj4" pod "bf858d52-3486-4db1-9a50-243e0ee683a5" (UID: "bf858d52-3486-4db1-9a50-243e0ee683a5"). InnerVolumeSpecName "kube-api-access-4whj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:50:54.938184 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.936682    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7mb\" (UniqueName: \"kubernetes.io/projected/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358-kube-api-access-9g7mb\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:54.938184 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:54.936718    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whj4\" (UniqueName: \"kubernetes.io/projected/bf858d52-3486-4db1-9a50-243e0ee683a5-kube-api-access-4whj4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:50:55.018964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.018891    5393 generic.go:356] "Generic (PLEG): container finished" podID="bf858d52-3486-4db1-9a50-243e0ee683a5" containerID="42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef" exitCode=0
Nov 03 10:50:55.018964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.018959    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"
Nov 03 10:50:55.019231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.018955    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl" event={"ID":"bf858d52-3486-4db1-9a50-243e0ee683a5","Type":"ContainerDied","Data":"42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef"}
Nov 03 10:50:55.019231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.019082    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl" event={"ID":"bf858d52-3486-4db1-9a50-243e0ee683a5","Type":"ContainerDied","Data":"f109aa1384ae6025a9d5017cda159284acbeed7aa2763d0f1c131d211c56030e"}
Nov 03 10:50:55.019231 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.019118    5393 scope.go:122] "RemoveContainer" containerID="42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef"
Nov 03 10:50:55.020622 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.020598    5393 generic.go:356] "Generic (PLEG): container finished" podID="1a7aa82b-84d0-4e85-a2d9-e0e9541d6358" containerID="95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989" exitCode=0
Nov 03 10:50:55.020733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.020635    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"
Nov 03 10:50:55.020733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.020692    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj" event={"ID":"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358","Type":"ContainerDied","Data":"95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989"}
Nov 03 10:50:55.020808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.020744    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj" event={"ID":"1a7aa82b-84d0-4e85-a2d9-e0e9541d6358","Type":"ContainerDied","Data":"511e96c80a30b7448b9da9bb6090e57023119fc863744510aaf2d3043e7a68c4"}
Nov 03 10:50:55.021106 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.021085    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:55.023141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.023123    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:50:55.032159 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.032139    5393 scope.go:122] "RemoveContainer" containerID="42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef"
Nov 03 10:50:55.032488 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:55.032468    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef\": container with ID starting with 42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef not found: ID does not exist" containerID="42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef"
Nov 03 10:50:55.032561 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.032496    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef"} err="failed to get container status \"42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef\": rpc error: code = NotFound desc = could not find container \"42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef\": container with ID starting with 42daccc39b315b67ac87b7d9681c500ae5186dba70efaecf8abb8ff222dec8ef not found: ID does not exist"
Nov 03 10:50:55.032561 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.032515    5393 scope.go:122] "RemoveContainer" containerID="95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989"
Nov 03 10:50:55.043667 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.043560    5393 scope.go:122] "RemoveContainer" containerID="95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989"
Nov 03 10:50:55.043865 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:50:55.043841    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989\": container with ID starting with 95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989 not found: ID does not exist" containerID="95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989"
Nov 03 10:50:55.043962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.043876    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989"} err="failed to get container status \"95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989\": rpc error: code = NotFound desc = could not find container \"95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989\": container with ID starting with 95acf9482fc95b1967b3e3eebf4c67efd90608517c0ceaa82836302e38e01989 not found: ID does not exist"
Nov 03 10:50:55.062499 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.062460    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"]
Nov 03 10:50:55.070274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.070245    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-5f4f85df9f-4x7nl"]
Nov 03 10:50:55.082436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.082397    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"]
Nov 03 10:50:55.087731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:55.087705    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-6c7dc97fc5-b7rlj"]
Nov 03 10:50:56.232117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:56.232087    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7aa82b-84d0-4e85-a2d9-e0e9541d6358" path="/var/lib/kubelet/pods/1a7aa82b-84d0-4e85-a2d9-e0e9541d6358/volumes"
Nov 03 10:50:56.232582 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:56.232567    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf858d52-3486-4db1-9a50-243e0ee683a5" path="/var/lib/kubelet/pods/bf858d52-3486-4db1-9a50-243e0ee683a5/volumes"
Nov 03 10:50:57.585357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.585322    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"]
Nov 03 10:50:57.586187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.586169    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1a7aa82b-84d0-4e85-a2d9-e0e9541d6358" containerName="payload-pre-processing"
Nov 03 10:50:57.586241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.586188    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7aa82b-84d0-4e85-a2d9-e0e9541d6358" containerName="payload-pre-processing"
Nov 03 10:50:57.586241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.586205    5393 cpu_manager.go:414] "containerMap: removing container" podUID="bf858d52-3486-4db1-9a50-243e0ee683a5" containerName="payload-processing"
Nov 03 10:50:57.586241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.586210    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf858d52-3486-4db1-9a50-243e0ee683a5" containerName="payload-processing"
Nov 03 10:50:57.586342 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.586333    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7aa82b-84d0-4e85-a2d9-e0e9541d6358" containerName="payload-pre-processing"
Nov 03 10:50:57.586374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.586346    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf858d52-3486-4db1-9a50-243e0ee683a5" containerName="payload-processing"
Nov 03 10:50:57.591289 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.591268    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:57.595655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.595631    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b3dc04\""
Nov 03 10:50:57.598510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.598485    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"]
Nov 03 10:50:57.617895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.617859    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"]
Nov 03 10:50:57.622064 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.622041    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:57.626548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.626504    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b3dc04-dockercfg-p6cd4\""
Nov 03 10:50:57.630703 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.630673    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"]
Nov 03 10:50:57.687524 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.687488    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6zz\" (UniqueName: \"kubernetes.io/projected/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-kube-api-access-wk6zz\") pod \"payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt\" (UID: \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:57.688017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.687986    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bdae2d-a392-49a5-85f9-4045f63cf92a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96\" (UID: \"73bdae2d-a392-49a5-85f9-4045f63cf92a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:57.688139 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.688092    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjqgm\" (UniqueName: \"kubernetes.io/projected/73bdae2d-a392-49a5-85f9-4045f63cf92a-kube-api-access-pjqgm\") pod \"payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96\" (UID: \"73bdae2d-a392-49a5-85f9-4045f63cf92a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:57.688205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.688173    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt\" (UID: \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:57.792943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.791850    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt\" (UID: \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:57.792943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.792003    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6zz\" (UniqueName: \"kubernetes.io/projected/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-kube-api-access-wk6zz\") pod \"payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt\" (UID: \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:57.792943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.792189    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bdae2d-a392-49a5-85f9-4045f63cf92a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96\" (UID: \"73bdae2d-a392-49a5-85f9-4045f63cf92a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:57.792943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.792248    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjqgm\" (UniqueName: \"kubernetes.io/projected/73bdae2d-a392-49a5-85f9-4045f63cf92a-kube-api-access-pjqgm\") pod \"payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96\" (UID: \"73bdae2d-a392-49a5-85f9-4045f63cf92a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:57.794954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.793626    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt\" (UID: \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:57.802941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.798967    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bdae2d-a392-49a5-85f9-4045f63cf92a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96\" (UID: \"73bdae2d-a392-49a5-85f9-4045f63cf92a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:57.839991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.824945    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjqgm\" (UniqueName: \"kubernetes.io/projected/73bdae2d-a392-49a5-85f9-4045f63cf92a-kube-api-access-pjqgm\") pod \"payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96\" (UID: \"73bdae2d-a392-49a5-85f9-4045f63cf92a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:57.839991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.827816    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6zz\" (UniqueName: \"kubernetes.io/projected/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-kube-api-access-wk6zz\") pod \"payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt\" (UID: \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:57.908792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.908761    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:57.947912 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:57.947887    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:58.356050 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.356011    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"]
Nov 03 10:50:58.410665 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:50:58.410622    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bdae2d_a392_49a5_85f9_4045f63cf92a.slice/crio-131cb1faeff89f23fb5b946598b249aec33a5aaa29a44909c60fa27ea2bc8ba3 WatchSource:0}: Error finding container 131cb1faeff89f23fb5b946598b249aec33a5aaa29a44909c60fa27ea2bc8ba3: Status 404 returned error can't find the container with id 131cb1faeff89f23fb5b946598b249aec33a5aaa29a44909c60fa27ea2bc8ba3
Nov 03 10:50:58.412481 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.412461    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"]
Nov 03 10:50:58.908413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.908381    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"]
Nov 03 10:50:58.912580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.912563    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:50:58.919724 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.919079    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-814dc0\""
Nov 03 10:50:58.919724 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.919088    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-814dc0-dockercfg-x5tld\""
Nov 03 10:50:58.922678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.922653    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"]
Nov 03 10:50:58.941155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.941125    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"]
Nov 03 10:50:58.945246 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.945223    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:50:58.956594 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:58.956568    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"]
Nov 03 10:50:59.088943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.088698    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszhj\" (UniqueName: \"kubernetes.io/projected/79284696-2fce-4cc6-9224-7e70b7831293-kube-api-access-nszhj\") pod \"payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h\" (UID: \"79284696-2fce-4cc6-9224-7e70b7831293\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:50:59.088943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.088769    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78r5q\" (UniqueName: \"kubernetes.io/projected/d10556d7-e0f7-451d-bba6-752192520de6-kube-api-access-78r5q\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4\" (UID: \"d10556d7-e0f7-451d-bba6-752192520de6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:50:59.089207 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.089042    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10556d7-e0f7-451d-bba6-752192520de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4\" (UID: \"d10556d7-e0f7-451d-bba6-752192520de6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:50:59.089207 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.089164    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79284696-2fce-4cc6-9224-7e70b7831293-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h\" (UID: \"79284696-2fce-4cc6-9224-7e70b7831293\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:50:59.173004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.171684    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt" event={"ID":"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c","Type":"ContainerStarted","Data":"0ff25068211d5839617531409d6f6a9fdbd88be1fd32faab8fa667b92d290e79"}
Nov 03 10:50:59.173004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.171746    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt" event={"ID":"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c","Type":"ContainerStarted","Data":"d4121b00b9d9b0316d0a8935d22886770c66ee44ff59c0ab6d38ac94c5123712"}
Nov 03 10:50:59.173004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.172758    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:50:59.194942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.192769    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10556d7-e0f7-451d-bba6-752192520de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4\" (UID: \"d10556d7-e0f7-451d-bba6-752192520de6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:50:59.194942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.192932    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79284696-2fce-4cc6-9224-7e70b7831293-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h\" (UID: \"79284696-2fce-4cc6-9224-7e70b7831293\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:50:59.194942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.193020    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszhj\" (UniqueName: \"kubernetes.io/projected/79284696-2fce-4cc6-9224-7e70b7831293-kube-api-access-nszhj\") pod \"payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h\" (UID: \"79284696-2fce-4cc6-9224-7e70b7831293\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:50:59.194942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.193061    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78r5q\" (UniqueName: \"kubernetes.io/projected/d10556d7-e0f7-451d-bba6-752192520de6-kube-api-access-78r5q\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4\" (UID: \"d10556d7-e0f7-451d-bba6-752192520de6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:50:59.194942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.194259    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10556d7-e0f7-451d-bba6-752192520de6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4\" (UID: \"d10556d7-e0f7-451d-bba6-752192520de6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:50:59.195496 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.195473    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79284696-2fce-4cc6-9224-7e70b7831293-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h\" (UID: \"79284696-2fce-4cc6-9224-7e70b7831293\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:50:59.238958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.227736    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78r5q\" (UniqueName: \"kubernetes.io/projected/d10556d7-e0f7-451d-bba6-752192520de6-kube-api-access-78r5q\") pod \"payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4\" (UID: \"d10556d7-e0f7-451d-bba6-752192520de6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:50:59.239226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.237626    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszhj\" (UniqueName: \"kubernetes.io/projected/79284696-2fce-4cc6-9224-7e70b7831293-kube-api-access-nszhj\") pod \"payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h\" (UID: \"79284696-2fce-4cc6-9224-7e70b7831293\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:50:59.257393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.243732    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:50:59.271943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.268663    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96" event={"ID":"73bdae2d-a392-49a5-85f9-4045f63cf92a","Type":"ContainerStarted","Data":"8a9a13aa5ed1a8c333a5398423d68b46467166fd1562f6799863a93a841c253f"}
Nov 03 10:50:59.274874 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.268841    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96" event={"ID":"73bdae2d-a392-49a5-85f9-4045f63cf92a","Type":"ContainerStarted","Data":"131cb1faeff89f23fb5b946598b249aec33a5aaa29a44909c60fa27ea2bc8ba3"}
Nov 03 10:50:59.275260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.275238    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:50:59.276201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.276184    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:50:59.301261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.290558    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt" podStartSLOduration=2.290540282 podStartE2EDuration="2.290540282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:59.228840442 +0000 UTC m=+1303.133694885" watchObservedRunningTime="2026-11-03 10:50:59.290540282 +0000 UTC m=+1303.195394723"
Nov 03 10:50:59.301261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.294727    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96" podStartSLOduration=2.294708752 podStartE2EDuration="2.294708752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:50:59.290359279 +0000 UTC m=+1303.195213720" watchObservedRunningTime="2026-11-03 10:50:59.294708752 +0000 UTC m=+1303.199563193"
Nov 03 10:50:59.915808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.915770    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"]
Nov 03 10:50:59.917270 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:50:59.917239    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79284696_2fce_4cc6_9224_7e70b7831293.slice/crio-b349dde1d48bd9d223c3fb67dc6b62b70f44d1a0b3967c02432ff4e2296408ba WatchSource:0}: Error finding container b349dde1d48bd9d223c3fb67dc6b62b70f44d1a0b3967c02432ff4e2296408ba: Status 404 returned error can't find the container with id b349dde1d48bd9d223c3fb67dc6b62b70f44d1a0b3967c02432ff4e2296408ba
Nov 03 10:50:59.940351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:50:59.940317    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"]
Nov 03 10:50:59.941274 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:50:59.941247    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10556d7_e0f7_451d_bba6_752192520de6.slice/crio-edc61d86f442948bc9d76ccee1bcf11339e43cabc49805bcc87c01d7a02e03cd WatchSource:0}: Error finding container edc61d86f442948bc9d76ccee1bcf11339e43cabc49805bcc87c01d7a02e03cd: Status 404 returned error can't find the container with id edc61d86f442948bc9d76ccee1bcf11339e43cabc49805bcc87c01d7a02e03cd
Nov 03 10:51:00.280450 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:00.280341    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4" event={"ID":"d10556d7-e0f7-451d-bba6-752192520de6","Type":"ContainerStarted","Data":"68fdeb6ff892392bf4f18d58598acc88e6f3fe3d43bfda1d4174add8ff37d742"}
Nov 03 10:51:00.280450 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:00.280412    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4" event={"ID":"d10556d7-e0f7-451d-bba6-752192520de6","Type":"ContainerStarted","Data":"edc61d86f442948bc9d76ccee1bcf11339e43cabc49805bcc87c01d7a02e03cd"}
Nov 03 10:51:00.280656 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:00.280560    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:51:00.294945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:00.292513    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h" event={"ID":"79284696-2fce-4cc6-9224-7e70b7831293","Type":"ContainerStarted","Data":"31194a66b74bf0f1db063106f217c168cc4d185879c77afc3d8223f024960be9"}
Nov 03 10:51:00.294945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:00.292571    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h" event={"ID":"79284696-2fce-4cc6-9224-7e70b7831293","Type":"ContainerStarted","Data":"b349dde1d48bd9d223c3fb67dc6b62b70f44d1a0b3967c02432ff4e2296408ba"}
Nov 03 10:51:00.294945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:00.292600    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:51:00.312943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:00.312148    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4" podStartSLOduration=2.312127276 podStartE2EDuration="2.312127276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:00.309530543 +0000 UTC m=+1304.214384984" watchObservedRunningTime="2026-11-03 10:51:00.312127276 +0000 UTC m=+1304.216981718"
Nov 03 10:51:00.331749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:00.331685    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h" podStartSLOduration=2.331666069 podStartE2EDuration="2.331666069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:00.32547193 +0000 UTC m=+1304.230326371" watchObservedRunningTime="2026-11-03 10:51:00.331666069 +0000 UTC m=+1304.236520512"
Nov 03 10:51:09.839992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:09.838872    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": read tcp 10.134.0.2:36276->10.134.0.109:15021: read: connection reset by peer" start-of-body=
Nov 03 10:51:09.839992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:09.838953    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": read tcp 10.134.0.2:36276->10.134.0.109:15021: read: connection reset by peer"
Nov 03 10:51:10.478587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:10.478558    5393 generic.go:356] "Generic (PLEG): container finished" podID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerID="a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263" exitCode=0
Nov 03 10:51:10.478775 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:10.478638    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" event={"ID":"a8c0b587-9ed2-4cbc-922a-74782922dc1a","Type":"ContainerDied","Data":"a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263"}
Nov 03 10:51:10.479263 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:10.479243    5393 scope.go:122] "RemoveContainer" containerID="a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263"
Nov 03 10:51:10.481121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:10.481091    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:10.481235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:10.481178    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:10.481235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:10.481225    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:11.347084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:11.347042    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:51:11.347649 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:11.347624    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:51:11.348207 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:11.348178    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:51:11.348686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:11.348664    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:51:11.497305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:11.497268    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" event={"ID":"a8c0b587-9ed2-4cbc-922a-74782922dc1a","Type":"ContainerStarted","Data":"50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198"}
Nov 03 10:51:11.621580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:11.621498    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:51:11.621736 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:11.621703    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:51:11.621786 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:11.621749    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 10:51:12.621690 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:12.621652    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:51:12.622194 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:12.621712    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 10:51:13.236561 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:51:13.236502    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:51:13.621111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:13.621003    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:51:13.621111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:13.621089    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 10:51:14.624441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:14.624406    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:51:14.625098 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:14.625067    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 10:51:15.621595 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:15.621553    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:51:15.621788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:15.621619    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 10:51:16.621592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:16.621555    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 10:51:16.622028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:16.621605    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 10:51:17.339071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.339037    5393 scope.go:122] "RemoveContainer" containerID="d51ba8d68eda65c7da5097a72a514ce00aca75a01b1e05b3bea50906867acede"
Nov 03 10:51:17.350013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.349982    5393 scope.go:122] "RemoveContainer" containerID="3297bf5f54f38425119df1bf8dba0148484e4aefed5a9a227354514b8e8ac2af"
Nov 03 10:51:17.359883 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.359861    5393 scope.go:122] "RemoveContainer" containerID="8424ee67cefbf50c5ff6ab0e04bfe55aa79527ec7c7e0af06d726e6cee657d4f"
Nov 03 10:51:17.369073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.369051    5393 scope.go:122] "RemoveContainer" containerID="85bc231166085ac91993076a26a895cfa69e6ffd1ad8b53a6e8738e649b5cf62"
Nov 03 10:51:17.379514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.379450    5393 scope.go:122] "RemoveContainer" containerID="62b838f765e65c988087aeb8dc5fd653710bc798bcb7ebcd961665897f6caabb"
Nov 03 10:51:17.390806 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.390781    5393 scope.go:122] "RemoveContainer" containerID="0028f67ce55d2eccb242f16761ce1d51119bc30fce4a0e1a4dda30bb3bff315a"
Nov 03 10:51:17.697956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.689210    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:51:17.697956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.690809    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:51:17.730038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:17.717485    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:51:29.034945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:29.033707    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"]
Nov 03 10:51:29.034945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:29.034049    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t" podUID="62a69b56-ebe9-4c52-aed2-aebc05bc4344" containerName="payload-pre-processing" containerID="cri-o://b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde" gracePeriod=30
Nov 03 10:51:29.975393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:29.975345    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:51:30.066946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.061065    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a69b56-ebe9-4c52-aed2-aebc05bc4344-kube-api-access-s4qjq\" (UniqueName: \"kubernetes.io/projected/62a69b56-ebe9-4c52-aed2-aebc05bc4344-kube-api-access-s4qjq\") pod \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\" (UID: \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\") "
Nov 03 10:51:30.066946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.061360    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a69b56-ebe9-4c52-aed2-aebc05bc4344-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a69b56-ebe9-4c52-aed2-aebc05bc4344-plugins-config-volume\") pod \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\" (UID: \"62a69b56-ebe9-4c52-aed2-aebc05bc4344\") "
Nov 03 10:51:30.066946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.062287    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a69b56-ebe9-4c52-aed2-aebc05bc4344-plugins-config-volume" pod "62a69b56-ebe9-4c52-aed2-aebc05bc4344" (UID: "62a69b56-ebe9-4c52-aed2-aebc05bc4344"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:30.084981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.077051    5393 generic.go:356] "Generic (PLEG): container finished" podID="62a69b56-ebe9-4c52-aed2-aebc05bc4344" containerID="b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde" exitCode=0
Nov 03 10:51:30.084981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.077265    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t" event={"ID":"62a69b56-ebe9-4c52-aed2-aebc05bc4344","Type":"ContainerDied","Data":"b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde"}
Nov 03 10:51:30.084981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.077370    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t" event={"ID":"62a69b56-ebe9-4c52-aed2-aebc05bc4344","Type":"ContainerDied","Data":"884a959dce5b124c6fde3fae8be8c2bc941c2037e0a16364d24caa44c7f06792"}
Nov 03 10:51:30.084981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.077408    5393 scope.go:122] "RemoveContainer" containerID="b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde"
Nov 03 10:51:30.084981 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.077665    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"
Nov 03 10:51:30.086017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.085968    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a69b56-ebe9-4c52-aed2-aebc05bc4344-kube-api-access-s4qjq" pod "62a69b56-ebe9-4c52-aed2-aebc05bc4344" (UID: "62a69b56-ebe9-4c52-aed2-aebc05bc4344"). InnerVolumeSpecName "kube-api-access-s4qjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:30.131779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.131762    5393 scope.go:122] "RemoveContainer" containerID="b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde"
Nov 03 10:51:30.132253 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:51:30.132201    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde\": container with ID starting with b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde not found: ID does not exist" containerID="b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde"
Nov 03 10:51:30.132359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.132244    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde"} err="failed to get container status \"b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde\": rpc error: code = NotFound desc = could not find container \"b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde\": container with ID starting with b6c3b79911405dab035faca2919cb549ace601a344b6a1c397fd8ce750bd8dde not found: ID does not exist"
Nov 03 10:51:30.163841 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.163807    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a69b56-ebe9-4c52-aed2-aebc05bc4344-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:30.163841 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.163838    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4qjq\" (UniqueName: \"kubernetes.io/projected/62a69b56-ebe9-4c52-aed2-aebc05bc4344-kube-api-access-s4qjq\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:30.411713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.411619    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"]
Nov 03 10:51:30.414877 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:30.414841    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a2697c-78f7fb465f-d845t"]
Nov 03 10:51:32.231887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:32.231854    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a69b56-ebe9-4c52-aed2-aebc05bc4344" path="/var/lib/kubelet/pods/62a69b56-ebe9-4c52-aed2-aebc05bc4344/volumes"
Nov 03 10:51:32.497604 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:32.497514    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"]
Nov 03 10:51:32.497792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:32.497762    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm" podUID="d451027a-a892-4f68-9da5-79ea8eab7d84" containerName="payload-processing" containerID="cri-o://19e4450a8552162c48498fc3e99898cd1f6a3f33bffd03e4d5011618d95d4a35" gracePeriod=30
Nov 03 10:51:33.119038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.100286    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"]
Nov 03 10:51:33.219028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.135301    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy" containerID="cri-o://57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722" gracePeriod=30
Nov 03 10:51:33.384290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.382016    5393 generic.go:356] "Generic (PLEG): container finished" podID="d451027a-a892-4f68-9da5-79ea8eab7d84" containerID="19e4450a8552162c48498fc3e99898cd1f6a3f33bffd03e4d5011618d95d4a35" exitCode=0
Nov 03 10:51:33.384290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.382180    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm" event={"ID":"d451027a-a892-4f68-9da5-79ea8eab7d84","Type":"ContainerDied","Data":"19e4450a8552162c48498fc3e99898cd1f6a3f33bffd03e4d5011618d95d4a35"}
Nov 03 10:51:33.480269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.454785    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"]
Nov 03 10:51:33.480269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.455066    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy" containerID="cri-o://b92e76d80c753dd695cbd9e7da2c6a69e7bd791fe0aef6a335d310674c7e1cbc" gracePeriod=30
Nov 03 10:51:33.482787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.482720    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:51:33.714599 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.714510    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d451027a-a892-4f68-9da5-79ea8eab7d84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d451027a-a892-4f68-9da5-79ea8eab7d84-plugins-config-volume\") pod \"d451027a-a892-4f68-9da5-79ea8eab7d84\" (UID: \"d451027a-a892-4f68-9da5-79ea8eab7d84\") "
Nov 03 10:51:33.714779 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.714617    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d451027a-a892-4f68-9da5-79ea8eab7d84-kube-api-access-dr8x9\" (UniqueName: \"kubernetes.io/projected/d451027a-a892-4f68-9da5-79ea8eab7d84-kube-api-access-dr8x9\") pod \"d451027a-a892-4f68-9da5-79ea8eab7d84\" (UID: \"d451027a-a892-4f68-9da5-79ea8eab7d84\") "
Nov 03 10:51:33.714932 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.714894    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d451027a-a892-4f68-9da5-79ea8eab7d84-plugins-config-volume" pod "d451027a-a892-4f68-9da5-79ea8eab7d84" (UID: "d451027a-a892-4f68-9da5-79ea8eab7d84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:33.715567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.715534    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d451027a-a892-4f68-9da5-79ea8eab7d84-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:33.717259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.717223    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d451027a-a892-4f68-9da5-79ea8eab7d84-kube-api-access-dr8x9" pod "d451027a-a892-4f68-9da5-79ea8eab7d84" (UID: "d451027a-a892-4f68-9da5-79ea8eab7d84"). InnerVolumeSpecName "kube-api-access-dr8x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:33.818386 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:33.817409    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr8x9\" (UniqueName: \"kubernetes.io/projected/d451027a-a892-4f68-9da5-79ea8eab7d84-kube-api-access-dr8x9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:34.394187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:34.394150    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm" event={"ID":"d451027a-a892-4f68-9da5-79ea8eab7d84","Type":"ContainerDied","Data":"370011d3ed244a9b394685a2f506aa894074a386efe0264ccf72c86c76e4b226"}
Nov 03 10:51:34.394649 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:34.394213    5393 scope.go:122] "RemoveContainer" containerID="19e4450a8552162c48498fc3e99898cd1f6a3f33bffd03e4d5011618d95d4a35"
Nov 03 10:51:34.394649 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:34.394360    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"
Nov 03 10:51:34.420954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:34.420907    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"]
Nov 03 10:51:34.421769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:34.421741    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4d9549-689b88f9dd-nvcmm"]
Nov 03 10:51:36.248191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:36.245885    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d451027a-a892-4f68-9da5-79ea8eab7d84" path="/var/lib/kubelet/pods/d451027a-a892-4f68-9da5-79ea8eab7d84/volumes"
Nov 03 10:51:36.324044 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:36.324008    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"]
Nov 03 10:51:36.324882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:36.324533    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy" containerID="cri-o://04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a" gracePeriod=30
Nov 03 10:51:37.694626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.694500    5393 generic.go:356] "Generic (PLEG): container finished" podID="ea688a05-d023-4385-8f9c-54ceac47a320" containerID="92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8" exitCode=0
Nov 03 10:51:37.695628 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.695347    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" event={"ID":"ea688a05-d023-4385-8f9c-54ceac47a320","Type":"ContainerDied","Data":"92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8"}
Nov 03 10:51:37.696517 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.696495    5393 scope.go:122] "RemoveContainer" containerID="92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8"
Nov 03 10:51:37.726945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.700703    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:37.837005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.830722    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:37.837005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.835322    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:37.891345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.877969    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"]
Nov 03 10:51:37.891345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.879166    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d451027a-a892-4f68-9da5-79ea8eab7d84" containerName="payload-processing"
Nov 03 10:51:37.891345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.879189    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d451027a-a892-4f68-9da5-79ea8eab7d84" containerName="payload-processing"
Nov 03 10:51:37.891345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.879332    5393 cpu_manager.go:414] "containerMap: removing container" podUID="62a69b56-ebe9-4c52-aed2-aebc05bc4344" containerName="payload-pre-processing"
Nov 03 10:51:37.891345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.879342    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a69b56-ebe9-4c52-aed2-aebc05bc4344" containerName="payload-pre-processing"
Nov 03 10:51:37.891345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.879512    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a69b56-ebe9-4c52-aed2-aebc05bc4344" containerName="payload-pre-processing"
Nov 03 10:51:37.891345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.879537    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d451027a-a892-4f68-9da5-79ea8eab7d84" containerName="payload-processing"
Nov 03 10:51:37.891879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.891357    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:37.922830 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:37.922795    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"]
Nov 03 10:51:38.063614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.063573    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.063877 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.063627    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6c4\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-kube-api-access-2c6c4\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.063877 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.063732    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.063877 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.063793    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.064072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.063879    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f79b86c-769b-47aa-a876-adc74caf37ef-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.064072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.063954    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.064072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.064003    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.064072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.064046    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.064217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.064164    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.165578 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.165540    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6c4\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-kube-api-access-2c6c4\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.165767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.165601    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.165767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.165656    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.165767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.165690    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f79b86c-769b-47aa-a876-adc74caf37ef-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.165767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.165735    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.165910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.165776    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.165910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.165838    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.166022 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.165967    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.166379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.166340    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.166742 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.166592    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-credential-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.166742 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.166681    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f79b86c-769b-47aa-a876-adc74caf37ef-istiod-ca-cert\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.167088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.167070    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-socket\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.167387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.167340    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-certs\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.167542 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.167521    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-data\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.169888 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.169861    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-envoy\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.170443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.170420    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-podinfo\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.204945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.192546    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-token\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.204945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.198538    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6c4\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-kube-api-access-2c6c4\") pod \"e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") " pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.305886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.305795    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:38.648807 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.648772    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"]
Nov 03 10:51:38.658946 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:51:38.656457    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f79b86c_769b_47aa_a876_adc74caf37ef.slice/crio-35d7b3cbecc2581c41c6c4b8b57050f0909922e802e77e18010c8b740f0e5d42 WatchSource:0}: Error finding container 35d7b3cbecc2581c41c6c4b8b57050f0909922e802e77e18010c8b740f0e5d42: Status 404 returned error can't find the container with id 35d7b3cbecc2581c41c6c4b8b57050f0909922e802e77e18010c8b740f0e5d42
Nov 03 10:51:38.662226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.660344    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:38.662226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.660435    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:38.662226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.660482    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:51:38.797409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.797379    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.838778    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" event={"ID":"ea688a05-d023-4385-8f9c-54ceac47a320","Type":"ContainerStarted","Data":"0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922"}
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841155    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istiod-ca-cert\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841198    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-token\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-token\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841255    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-envoy\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841289    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-socket\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841327    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-data\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841405    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-kube-api-access-wjlbj\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-kube-api-access-wjlbj\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841494    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-podinfo\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841564    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-credential-socket\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.841953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.841707    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-certs\") pod \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\" (UID: \"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0\") "
Nov 03 10:51:38.874005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.843892    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-socket" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:38.874005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.844353    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istiod-ca-cert" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:38.874005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.844611    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-credential-socket" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:38.874005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.844771    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-certs" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:38.874005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.845971    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-data" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:38.874005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.850302    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" event={"ID":"5f79b86c-769b-47aa-a876-adc74caf37ef","Type":"ContainerStarted","Data":"35d7b3cbecc2581c41c6c4b8b57050f0909922e802e77e18010c8b740f0e5d42"}
Nov 03 10:51:38.898800 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.898578    5393 generic.go:356] "Generic (PLEG): container finished" podID="03c6b93a-430d-44ac-9d84-335e693275af" containerID="b92e76d80c753dd695cbd9e7da2c6a69e7bd791fe0aef6a335d310674c7e1cbc" exitCode=0
Nov 03 10:51:38.898800 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.898709    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" event={"ID":"03c6b93a-430d-44ac-9d84-335e693275af","Type":"ContainerDied","Data":"b92e76d80c753dd695cbd9e7da2c6a69e7bd791fe0aef6a335d310674c7e1cbc"}
Nov 03 10:51:38.928694 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.928657    5393 generic.go:356] "Generic (PLEG): container finished" podID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerID="57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722" exitCode=0
Nov 03 10:51:38.928968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.928942    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" event={"ID":"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0","Type":"ContainerDied","Data":"57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722"}
Nov 03 10:51:38.929125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.929109    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj" event={"ID":"09c1f4ec-83c5-4332-a2c3-53360d2ab8e0","Type":"ContainerDied","Data":"75038b2a584ab6d7500dfeea187cf2d410843987c2b7743b7192f22fb8eb24a1"}
Nov 03 10:51:38.929260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.929247    5393 scope.go:122] "RemoveContainer" containerID="57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722"
Nov 03 10:51:38.929525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.929508    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"
Nov 03 10:51:38.950762 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.950733    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:38.975053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.975006    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:38.975473 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.975444    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:38.975605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.975590    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:38.975702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:38.975688    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.049465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.029220    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-token" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:39.049465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.030738    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-podinfo" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:39.049465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.035262    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-kube-api-access-wjlbj" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "kube-api-access-wjlbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:39.096556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.094873    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.096556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.094906    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjlbj\" (UniqueName: \"kubernetes.io/projected/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-kube-api-access-wjlbj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.096556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.094942    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.104690 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.104540    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-envoy" pod "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" (UID: "09c1f4ec-83c5-4332-a2c3-53360d2ab8e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:39.182139 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.182050    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:51:39.206947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.201524    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.207590 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.202028    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 10:51:39.207805 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.207776    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 10:51:39.259346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.259312    5393 scope.go:122] "RemoveContainer" containerID="57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722"
Nov 03 10:51:39.259819 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:51:39.259790    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722\": container with ID starting with 57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722 not found: ID does not exist" containerID="57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722"
Nov 03 10:51:39.259906 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.259844    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722"} err="failed to get container status \"57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722\": rpc error: code = NotFound desc = could not find container \"57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722\": container with ID starting with 57aff0a9e79b3701f37a5705c5db287c36ac470816ffc997928cfa78ba6d4722 not found: ID does not exist"
Nov 03 10:51:39.291119 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.291084    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"]
Nov 03 10:51:39.303146 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.302680    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-b2mfj"]
Nov 03 10:51:39.640851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.640824    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:51:39.726619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.726569    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03c6b93a-430d-44ac-9d84-335e693275af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6b93a-430d-44ac-9d84-335e693275af-istio-podinfo\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.726994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.726958    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-istio-token\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-istio-token\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.727259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.727240    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-data\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.728692 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.727389    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-envoy\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.728692 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.727427    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-certs\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.728692 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.727615    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-credential-socket\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.728692 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.727670    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c6b93a-430d-44ac-9d84-335e693275af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6b93a-430d-44ac-9d84-335e693275af-istiod-ca-cert\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.728692 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.727720    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-kube-api-access-zjtjd\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-kube-api-access-zjtjd\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.728692 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.727835    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-socket\") pod \"03c6b93a-430d-44ac-9d84-335e693275af\" (UID: \"03c6b93a-430d-44ac-9d84-335e693275af\") "
Nov 03 10:51:39.729160 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.729137    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-socket" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:39.729499 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.729480    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-certs" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:39.729774 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.729749    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c6b93a-430d-44ac-9d84-335e693275af-istiod-ca-cert" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:39.730074 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.730048    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-credential-socket" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:39.730716 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.730694    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-data" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:39.735625 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.735595    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-envoy" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:39.738480 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.738438    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-kube-api-access-zjtjd" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "kube-api-access-zjtjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:39.738744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.738716    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-istio-token" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:39.738997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.738902    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03c6b93a-430d-44ac-9d84-335e693275af-istio-podinfo" pod "03c6b93a-430d-44ac-9d84-335e693275af" (UID: "03c6b93a-430d-44ac-9d84-335e693275af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:39.842553 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.842500    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.843247 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.843226    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6b93a-430d-44ac-9d84-335e693275af-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.843372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.843358    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjtjd\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-kube-api-access-zjtjd\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.843471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.843459    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.843555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.843544    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6b93a-430d-44ac-9d84-335e693275af-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.843642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.843630    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c6b93a-430d-44ac-9d84-335e693275af-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.843730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.843717    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.843842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.843803    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.843842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.843822    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6b93a-430d-44ac-9d84-335e693275af-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:39.999999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:39.991435    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" event={"ID":"5f79b86c-769b-47aa-a876-adc74caf37ef","Type":"ContainerStarted","Data":"92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279"}
Nov 03 10:51:40.004945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.001664    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"
Nov 03 10:51:40.004945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.004026    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9" event={"ID":"03c6b93a-430d-44ac-9d84-335e693275af","Type":"ContainerDied","Data":"48c891324b268ce6081b46d8ae42039c552dc777b60c1688b6e7adb03a0ade18"}
Nov 03 10:51:40.004945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.004107    5393 scope.go:122] "RemoveContainer" containerID="b92e76d80c753dd695cbd9e7da2c6a69e7bd791fe0aef6a335d310674c7e1cbc"
Nov 03 10:51:40.024241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.020457    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podStartSLOduration=3.020436922 podStartE2EDuration="3.020436922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:40.013717481 +0000 UTC m=+1343.918571923" watchObservedRunningTime="2026-11-03 10:51:40.020436922 +0000 UTC m=+1343.925291364"
Nov 03 10:51:40.031161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.030349    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:51:40.061641 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.061581    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"]
Nov 03 10:51:40.068075 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.067057    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-hmvw9"]
Nov 03 10:51:40.139353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.139317    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"]
Nov 03 10:51:40.148305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.148265    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy" containerID="cri-o://9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0" gracePeriod=30
Nov 03 10:51:40.427036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.418340    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 10:51:40.427036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.418399    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 10:51:40.427036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.418573    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 10:51:40.427036 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.418622    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 10:51:40.498379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.492351    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c6b93a-430d-44ac-9d84-335e693275af" path="/var/lib/kubelet/pods/03c6b93a-430d-44ac-9d84-335e693275af/volumes"
Nov 03 10:51:40.498379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.493339    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" path="/var/lib/kubelet/pods/09c1f4ec-83c5-4332-a2c3-53360d2ab8e0/volumes"
Nov 03 10:51:40.498379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:40.493950    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:41.178843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.178814    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 10:51:41.179372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.179020    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 10:51:41.324944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.308418    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 10:51:41.324944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.308500    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 10:51:41.776267 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.776238    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:51:41.877279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877244    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-envoy\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.877471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877295    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-credential-socket\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.877471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877343    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-data\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.877471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877384    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-token\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-token\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.877471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877446    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-certs\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.877713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877480    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-podinfo\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.877713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877513    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-socket\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.877713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877536    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-credential-socket" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:41.877713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877547    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istiod-ca-cert\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.877976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877775    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-certs" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:41.877976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877900    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-data" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:41.877976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.877966    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istiod-ca-cert" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:41.878463 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.878439    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-socket" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:41.878571 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.878474    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-kube-api-access-ltz46\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-kube-api-access-ltz46\") pod \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\" (UID: \"ce655ad0-1a5a-4615-863a-9fd43685b4bb\") "
Nov 03 10:51:41.878997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.878975    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:41.878997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.878998    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:41.879160 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.879007    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:41.879160 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.879015    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:41.879160 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.879023    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:41.880868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.880847    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-token" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:41.896972 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.892019    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-podinfo" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:41.937946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.915012    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-kube-api-access-ltz46" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "kube-api-access-ltz46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:41.938319 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.938287    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-envoy" pod "ce655ad0-1a5a-4615-863a-9fd43685b4bb" (UID: "ce655ad0-1a5a-4615-863a-9fd43685b4bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:41.980569 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.980535    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:41.980569 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.980567    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:41.980776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.980581    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce655ad0-1a5a-4615-863a-9fd43685b4bb-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:41.980776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:41.980595    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltz46\" (UniqueName: \"kubernetes.io/projected/ce655ad0-1a5a-4615-863a-9fd43685b4bb-kube-api-access-ltz46\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:42.144301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.144273    5393 generic.go:356] "Generic (PLEG): container finished" podID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerID="04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a" exitCode=0
Nov 03 10:51:42.144511 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.144497    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"
Nov 03 10:51:42.144993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.144429    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" event={"ID":"ce655ad0-1a5a-4615-863a-9fd43685b4bb","Type":"ContainerDied","Data":"04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a"}
Nov 03 10:51:42.145117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.145065    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t" event={"ID":"ce655ad0-1a5a-4615-863a-9fd43685b4bb","Type":"ContainerDied","Data":"b8cfa78174219bc112e1f99395b4db5dc05ea442e1ed5fb17b853d3a65cbedaf"}
Nov 03 10:51:42.145117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.145105    5393 scope.go:122] "RemoveContainer" containerID="04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a"
Nov 03 10:51:42.157048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.157032    5393 scope.go:122] "RemoveContainer" containerID="287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0"
Nov 03 10:51:42.173428 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.173408    5393 scope.go:122] "RemoveContainer" containerID="04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a"
Nov 03 10:51:42.173900 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:51:42.173879    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a\": container with ID starting with 04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a not found: ID does not exist" containerID="04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a"
Nov 03 10:51:42.174040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.173909    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a"} err="failed to get container status \"04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a\": rpc error: code = NotFound desc = could not find container \"04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a\": container with ID starting with 04bd54b7ab90648119733d07543c2e6cb4003a9c5fde5dd864498400c4085f2a not found: ID does not exist"
Nov 03 10:51:42.174040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.173945    5393 scope.go:122] "RemoveContainer" containerID="287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0"
Nov 03 10:51:42.174371 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:51:42.174354    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0\": container with ID starting with 287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0 not found: ID does not exist" containerID="287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0"
Nov 03 10:51:42.174457 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.174374    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0"} err="failed to get container status \"287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0\": rpc error: code = NotFound desc = could not find container \"287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0\": container with ID starting with 287def3ef9210d2f1b8b32e3e0947d5e75ed28532a0e56433c0bc76ee9a943a0 not found: ID does not exist"
Nov 03 10:51:42.175290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.175276    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"]
Nov 03 10:51:42.178333 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.178311    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a2697c-openshift-default-785ff98597-d6s6t"]
Nov 03 10:51:42.178673 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.178655    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 10:51:42.178798 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.178779    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 10:51:42.250879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.245048    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" path="/var/lib/kubelet/pods/ce655ad0-1a5a-4615-863a-9fd43685b4bb/volumes"
Nov 03 10:51:42.322944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.317371    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 10:51:42.322944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:42.317432    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 10:51:43.179485 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.179435    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 10:51:43.179684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.179506    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 10:51:43.238672 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:51:43.238614    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:51:43.310106 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.309569    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 10:51:43.310106 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.309631    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.441439    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"]
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442373    5393 cpu_manager.go:414] "containerMap: removing container" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442392    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442418    5393 cpu_manager.go:414] "containerMap: removing container" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442425    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442455    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442462    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442482    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442488    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442636    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c1f4ec-83c5-4332-a2c3-53360d2ab8e0" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442648    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c6b93a-430d-44ac-9d84-335e693275af" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442662    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy"
Nov 03 10:51:43.445593 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.442673    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce655ad0-1a5a-4615-863a-9fd43685b4bb" containerName="istio-proxy"
Nov 03 10:51:43.455831 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.455806    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:43.461996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.459065    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"]
Nov 03 10:51:43.462155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.462043    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"]
Nov 03 10:51:43.469897 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.469784    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-835861-dockercfg-qml2p\""
Nov 03 10:51:43.469897 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.469849    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-835861\""
Nov 03 10:51:43.471315 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.471131    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:43.474436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.474397    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"]
Nov 03 10:51:43.794061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.736818    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f3652-eb44-4271-acf2-076be8e109bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc\" (UID: \"829f3652-eb44-4271-acf2-076be8e109bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:43.794061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.736994    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtbh\" (UniqueName: \"kubernetes.io/projected/829f3652-eb44-4271-acf2-076be8e109bf-kube-api-access-gbtbh\") pod \"payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc\" (UID: \"829f3652-eb44-4271-acf2-076be8e109bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:43.794061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.737145    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc2511-2821-4359-a513-63a2f2aaa5b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p\" (UID: \"5afc2511-2821-4359-a513-63a2f2aaa5b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:43.794061 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.737208    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttp6c\" (UniqueName: \"kubernetes.io/projected/5afc2511-2821-4359-a513-63a2f2aaa5b6-kube-api-access-ttp6c\") pod \"payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p\" (UID: \"5afc2511-2821-4359-a513-63a2f2aaa5b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:43.843461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.843429    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc2511-2821-4359-a513-63a2f2aaa5b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p\" (UID: \"5afc2511-2821-4359-a513-63a2f2aaa5b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:43.843807 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.843782    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttp6c\" (UniqueName: \"kubernetes.io/projected/5afc2511-2821-4359-a513-63a2f2aaa5b6-kube-api-access-ttp6c\") pod \"payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p\" (UID: \"5afc2511-2821-4359-a513-63a2f2aaa5b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:43.868286 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.868247    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f3652-eb44-4271-acf2-076be8e109bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc\" (UID: \"829f3652-eb44-4271-acf2-076be8e109bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:43.869972 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.869947    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtbh\" (UniqueName: \"kubernetes.io/projected/829f3652-eb44-4271-acf2-076be8e109bf-kube-api-access-gbtbh\") pod \"payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc\" (UID: \"829f3652-eb44-4271-acf2-076be8e109bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:43.877343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.869408    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f3652-eb44-4271-acf2-076be8e109bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc\" (UID: \"829f3652-eb44-4271-acf2-076be8e109bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:43.877807 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.844529    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc2511-2821-4359-a513-63a2f2aaa5b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p\" (UID: \"5afc2511-2821-4359-a513-63a2f2aaa5b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:43.972941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.967958    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttp6c\" (UniqueName: \"kubernetes.io/projected/5afc2511-2821-4359-a513-63a2f2aaa5b6-kube-api-access-ttp6c\") pod \"payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p\" (UID: \"5afc2511-2821-4359-a513-63a2f2aaa5b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:43.973847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:43.973817    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtbh\" (UniqueName: \"kubernetes.io/projected/829f3652-eb44-4271-acf2-076be8e109bf-kube-api-access-gbtbh\") pod \"payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc\" (UID: \"829f3652-eb44-4271-acf2-076be8e109bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:44.096031 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:44.083611    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:44.176456 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:44.169175    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:44.244094 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:44.243626    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 10:51:44.244094 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:44.243684    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 10:51:44.331944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:44.310895    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 10:51:44.331944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:44.310994    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 10:51:44.648004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:44.645989    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"]
Nov 03 10:51:44.651789 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:44.651372    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"]
Nov 03 10:51:45.202099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.202054    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 10:51:45.202290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.202124    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 10:51:45.323978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.320362    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 10:51:45.323978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.320423    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 10:51:45.329889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.329850    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p" event={"ID":"5afc2511-2821-4359-a513-63a2f2aaa5b6","Type":"ContainerStarted","Data":"53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21"}
Nov 03 10:51:45.330052 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.329941    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p" event={"ID":"5afc2511-2821-4359-a513-63a2f2aaa5b6","Type":"ContainerStarted","Data":"a6f701de826808e43a434bf5ff573129bb28a44fde6c03811f5917fd747b10fb"}
Nov 03 10:51:45.330744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.330723    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:45.333111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.332990    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc" event={"ID":"829f3652-eb44-4271-acf2-076be8e109bf","Type":"ContainerStarted","Data":"e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189"}
Nov 03 10:51:45.333111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.333028    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc" event={"ID":"829f3652-eb44-4271-acf2-076be8e109bf","Type":"ContainerStarted","Data":"66b040cac5cd50887755da9b24e10e0ad56c9bd0454ec13753fedd6e2b75f31c"}
Nov 03 10:51:45.333626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.333610    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:51:45.354184 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.353411    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p" podStartSLOduration=2.353389938 podStartE2EDuration="2.353389938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:45.35105197 +0000 UTC m=+1349.255906411" watchObservedRunningTime="2026-11-03 10:51:45.353389938 +0000 UTC m=+1349.258244379"
Nov 03 10:51:45.369109 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.366495    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc" podStartSLOduration=2.366475755 podStartE2EDuration="2.366475755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:45.365776318 +0000 UTC m=+1349.270630759" watchObservedRunningTime="2026-11-03 10:51:45.366475755 +0000 UTC m=+1349.271330196"
Nov 03 10:51:45.720213 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.720186    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:51:45.788946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.788886    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-credential-socket\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.789272 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.789250    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-socket\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.789477 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.789460    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-token\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-token\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.789661 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.789645    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-certs\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.789874 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.789857    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8395d93-9b56-4476-a706-3e00b3847ef6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8395d93-9b56-4476-a706-3e00b3847ef6-istiod-ca-cert\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.790081 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.790063    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-podinfo\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.790195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.790179    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-kube-api-access-jhhgt\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-kube-api-access-jhhgt\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.790454 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.790439    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-envoy\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.790577 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.790563    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-data\") pod \"d8395d93-9b56-4476-a706-3e00b3847ef6\" (UID: \"d8395d93-9b56-4476-a706-3e00b3847ef6\") "
Nov 03 10:51:45.798186 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.796405    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-credential-socket" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:45.798186 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.797218    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8395d93-9b56-4476-a706-3e00b3847ef6-istiod-ca-cert" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:51:45.798867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.798845    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-socket" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:45.799104 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.799074    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-data" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:45.799442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.799375    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-certs" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:45.819978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.819106    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-envoy" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:51:45.819978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.819569    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-kube-api-access-jhhgt" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "kube-api-access-jhhgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:45.820286 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.820245    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-token" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:51:45.822478 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.822441    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-podinfo" pod "d8395d93-9b56-4476-a706-3e00b3847ef6" (UID: "d8395d93-9b56-4476-a706-3e00b3847ef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900389    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8395d93-9b56-4476-a706-3e00b3847ef6-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900424    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900440    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhhgt\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-kube-api-access-jhhgt\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900455    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900468    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900482    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900496    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900509    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8395d93-9b56-4476-a706-3e00b3847ef6-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:45.903942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:45.900523    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8395d93-9b56-4476-a706-3e00b3847ef6-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:51:46.184657 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.184626    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:51:46.186340 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.186313    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:51:46.306781 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.306748    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 10:51:46.306985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.306798    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 10:51:46.344941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.344044    5393 generic.go:356] "Generic (PLEG): container finished" podID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerID="9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0" exitCode=0
Nov 03 10:51:46.344941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.344112    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"
Nov 03 10:51:46.344941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.344164    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" event={"ID":"d8395d93-9b56-4476-a706-3e00b3847ef6","Type":"ContainerDied","Data":"9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0"}
Nov 03 10:51:46.344941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.344230    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj" event={"ID":"d8395d93-9b56-4476-a706-3e00b3847ef6","Type":"ContainerDied","Data":"7e1db6961d04cd7032a424441d1e88e91f155e852e7823c06485ac2be6cd677c"}
Nov 03 10:51:46.344941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.344258    5393 scope.go:122] "RemoveContainer" containerID="9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0"
Nov 03 10:51:46.360150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.360125    5393 scope.go:122] "RemoveContainer" containerID="210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da"
Nov 03 10:51:46.369825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.369796    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"]
Nov 03 10:51:46.373659 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.373633    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4d9549-openshift-default-6c4cfd6bf-m7njj"]
Nov 03 10:51:46.379098 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.379072    5393 scope.go:122] "RemoveContainer" containerID="9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0"
Nov 03 10:51:46.379506 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:51:46.379486    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0\": container with ID starting with 9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0 not found: ID does not exist" containerID="9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0"
Nov 03 10:51:46.379585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.379517    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0"} err="failed to get container status \"9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0\": rpc error: code = NotFound desc = could not find container \"9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0\": container with ID starting with 9acfb705a3bc69e90b2cc75755a09f993d0668c0215f2e422b45dfa3974dd7e0 not found: ID does not exist"
Nov 03 10:51:46.379585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.379536    5393 scope.go:122] "RemoveContainer" containerID="210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da"
Nov 03 10:51:46.379869 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:51:46.379849    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da\": container with ID starting with 210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da not found: ID does not exist" containerID="210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da"
Nov 03 10:51:46.379947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:46.379879    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da"} err="failed to get container status \"210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da\": rpc error: code = NotFound desc = could not find container \"210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da\": container with ID starting with 210a3a74073ee1f54d4941c1db5397f50f8d5a68eb8f0f2c6b1c2341dbcf43da not found: ID does not exist"
Nov 03 10:51:47.329962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:47.322245    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:47.329962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:47.323931    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:47.329962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:47.328306    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:51:48.215813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.215778    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"]
Nov 03 10:51:48.216434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.216420    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy"
Nov 03 10:51:48.216434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.216435    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy"
Nov 03 10:51:48.216522 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.216450    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy"
Nov 03 10:51:48.216522 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.216457    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy"
Nov 03 10:51:48.216585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.216547    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy"
Nov 03 10:51:48.216585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.216555    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" containerName="istio-proxy"
Nov 03 10:51:48.221408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.221393    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:48.226869 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.226846    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f81a79\""
Nov 03 10:51:48.227084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.226850    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f81a79-dockercfg-l87sn\""
Nov 03 10:51:48.239520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.239485    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8395d93-9b56-4476-a706-3e00b3847ef6" path="/var/lib/kubelet/pods/d8395d93-9b56-4476-a706-3e00b3847ef6/volumes"
Nov 03 10:51:48.242793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.242764    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"]
Nov 03 10:51:48.247405 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.247380    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"]
Nov 03 10:51:48.255230 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.255207    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:48.261597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.261570    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"]
Nov 03 10:51:48.406948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.406480    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99kss\" (UniqueName: \"kubernetes.io/projected/c01721d7-ef7a-40d9-af85-62b1bfa78f97-kube-api-access-99kss\") pod \"payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c\" (UID: \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:48.406948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.406556    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zvm\" (UniqueName: \"kubernetes.io/projected/58bad0a0-d5f1-469d-b11a-26af1f55fe16-kube-api-access-46zvm\") pod \"payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr\" (UID: \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:48.406948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.406631    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01721d7-ef7a-40d9-af85-62b1bfa78f97-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c\" (UID: \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:48.406948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.406748    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bad0a0-d5f1-469d-b11a-26af1f55fe16-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr\" (UID: \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:48.512997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.508977    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bad0a0-d5f1-469d-b11a-26af1f55fe16-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr\" (UID: \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:48.512997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.509163    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99kss\" (UniqueName: \"kubernetes.io/projected/c01721d7-ef7a-40d9-af85-62b1bfa78f97-kube-api-access-99kss\") pod \"payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c\" (UID: \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:48.512997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.509215    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zvm\" (UniqueName: \"kubernetes.io/projected/58bad0a0-d5f1-469d-b11a-26af1f55fe16-kube-api-access-46zvm\") pod \"payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr\" (UID: \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:48.512997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.509304    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01721d7-ef7a-40d9-af85-62b1bfa78f97-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c\" (UID: \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:48.512997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.510397    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01721d7-ef7a-40d9-af85-62b1bfa78f97-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c\" (UID: \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:48.530525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.522208    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bad0a0-d5f1-469d-b11a-26af1f55fe16-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr\" (UID: \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:48.538540 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.538476    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zvm\" (UniqueName: \"kubernetes.io/projected/58bad0a0-d5f1-469d-b11a-26af1f55fe16-kube-api-access-46zvm\") pod \"payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr\" (UID: \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:48.560446 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.558618    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:48.562883 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.562852    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99kss\" (UniqueName: \"kubernetes.io/projected/c01721d7-ef7a-40d9-af85-62b1bfa78f97-kube-api-access-99kss\") pod \"payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c\" (UID: \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:48.573950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.573873    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:48.898796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.898769    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"]
Nov 03 10:51:48.899960 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:51:48.899911    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bad0a0_d5f1_469d_b11a_26af1f55fe16.slice/crio-eaa1b6b5ba3dc8c277d06c3e238cd946ad351a4b53ce38bd7db6627900fb3843 WatchSource:0}: Error finding container eaa1b6b5ba3dc8c277d06c3e238cd946ad351a4b53ce38bd7db6627900fb3843: Status 404 returned error can't find the container with id eaa1b6b5ba3dc8c277d06c3e238cd946ad351a4b53ce38bd7db6627900fb3843
Nov 03 10:51:48.927120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:48.927097    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"]
Nov 03 10:51:48.936235 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:51:48.936208    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01721d7_ef7a_40d9_af85_62b1bfa78f97.slice/crio-c12a86764ff652defdc4b5529764ca46d25acc98c139505ef0979c3111a1f373 WatchSource:0}: Error finding container c12a86764ff652defdc4b5529764ca46d25acc98c139505ef0979c3111a1f373: Status 404 returned error can't find the container with id c12a86764ff652defdc4b5529764ca46d25acc98c139505ef0979c3111a1f373
Nov 03 10:51:49.470199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:49.470161    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c" event={"ID":"c01721d7-ef7a-40d9-af85-62b1bfa78f97","Type":"ContainerStarted","Data":"c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b"}
Nov 03 10:51:49.470672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:49.470230    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c" event={"ID":"c01721d7-ef7a-40d9-af85-62b1bfa78f97","Type":"ContainerStarted","Data":"c12a86764ff652defdc4b5529764ca46d25acc98c139505ef0979c3111a1f373"}
Nov 03 10:51:49.470672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:49.470360    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:51:49.472190 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:49.472166    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr" event={"ID":"58bad0a0-d5f1-469d-b11a-26af1f55fe16","Type":"ContainerStarted","Data":"8b3024034c9b419a9bd1ea7f570fcd5802c162855635ba426b90b096f504522f"}
Nov 03 10:51:49.472302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:49.472213    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:51:49.472302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:49.472229    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr" event={"ID":"58bad0a0-d5f1-469d-b11a-26af1f55fe16","Type":"ContainerStarted","Data":"eaa1b6b5ba3dc8c277d06c3e238cd946ad351a4b53ce38bd7db6627900fb3843"}
Nov 03 10:51:49.486357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:49.486308    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c" podStartSLOduration=1.486295502 podStartE2EDuration="1.486295502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:49.485309613 +0000 UTC m=+1353.390164055" watchObservedRunningTime="2026-11-03 10:51:49.486295502 +0000 UTC m=+1353.391149943"
Nov 03 10:51:49.501544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:49.501497    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr" podStartSLOduration=1.501482868 podStartE2EDuration="1.501482868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:51:49.500049433 +0000 UTC m=+1353.404903887" watchObservedRunningTime="2026-11-03 10:51:49.501482868 +0000 UTC m=+1353.406337308"
Nov 03 10:51:57.371695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:57.371657    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:51:57.372161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:51:57.371728    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:52:00.515943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:00.515108    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:52:00.537943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:00.531153    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:52:01.028945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.020011    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"]
Nov 03 10:52:01.028945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.020251    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h" podUID="79284696-2fce-4cc6-9224-7e70b7831293" containerName="payload-processing" containerID="cri-o://31194a66b74bf0f1db063106f217c168cc4d185879c77afc3d8223f024960be9" gracePeriod=30
Nov 03 10:52:01.033357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.032896    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"]
Nov 03 10:52:01.033357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.033172    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4" podUID="d10556d7-e0f7-451d-bba6-752192520de6" containerName="payload-pre-processing" containerID="cri-o://68fdeb6ff892392bf4f18d58598acc88e6f3fe3d43bfda1d4174add8ff37d742" gracePeriod=30
Nov 03 10:52:01.349405 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.349355    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h" podUID="79284696-2fce-4cc6-9224-7e70b7831293" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.118:9004: connect: connection refused"
Nov 03 10:52:01.352279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.352005    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4" podUID="d10556d7-e0f7-451d-bba6-752192520de6" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.117:9004: connect: connection refused"
Nov 03 10:52:01.747087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.740852    5393 generic.go:356] "Generic (PLEG): container finished" podID="d10556d7-e0f7-451d-bba6-752192520de6" containerID="68fdeb6ff892392bf4f18d58598acc88e6f3fe3d43bfda1d4174add8ff37d742" exitCode=0
Nov 03 10:52:01.747087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.741041    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4" event={"ID":"d10556d7-e0f7-451d-bba6-752192520de6","Type":"ContainerDied","Data":"68fdeb6ff892392bf4f18d58598acc88e6f3fe3d43bfda1d4174add8ff37d742"}
Nov 03 10:52:01.747087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.745700    5393 generic.go:356] "Generic (PLEG): container finished" podID="79284696-2fce-4cc6-9224-7e70b7831293" containerID="31194a66b74bf0f1db063106f217c168cc4d185879c77afc3d8223f024960be9" exitCode=0
Nov 03 10:52:01.747087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.745735    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h" event={"ID":"79284696-2fce-4cc6-9224-7e70b7831293","Type":"ContainerDied","Data":"31194a66b74bf0f1db063106f217c168cc4d185879c77afc3d8223f024960be9"}
Nov 03 10:52:01.819553 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.819250    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:52:01.989142 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.986635    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10556d7-e0f7-451d-bba6-752192520de6-kube-api-access-78r5q\" (UniqueName: \"kubernetes.io/projected/d10556d7-e0f7-451d-bba6-752192520de6-kube-api-access-78r5q\") pod \"d10556d7-e0f7-451d-bba6-752192520de6\" (UID: \"d10556d7-e0f7-451d-bba6-752192520de6\") "
Nov 03 10:52:01.989142 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.986785    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10556d7-e0f7-451d-bba6-752192520de6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10556d7-e0f7-451d-bba6-752192520de6-plugins-config-volume\") pod \"d10556d7-e0f7-451d-bba6-752192520de6\" (UID: \"d10556d7-e0f7-451d-bba6-752192520de6\") "
Nov 03 10:52:01.990677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.990641    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10556d7-e0f7-451d-bba6-752192520de6-plugins-config-volume" pod "d10556d7-e0f7-451d-bba6-752192520de6" (UID: "d10556d7-e0f7-451d-bba6-752192520de6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:01.991201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:01.991182    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10556d7-e0f7-451d-bba6-752192520de6-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:02.008697 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.007300    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10556d7-e0f7-451d-bba6-752192520de6-kube-api-access-78r5q" pod "d10556d7-e0f7-451d-bba6-752192520de6" (UID: "d10556d7-e0f7-451d-bba6-752192520de6"). InnerVolumeSpecName "kube-api-access-78r5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:02.015904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.015875    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:52:02.092108 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.092071    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79284696-2fce-4cc6-9224-7e70b7831293-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79284696-2fce-4cc6-9224-7e70b7831293-plugins-config-volume\") pod \"79284696-2fce-4cc6-9224-7e70b7831293\" (UID: \"79284696-2fce-4cc6-9224-7e70b7831293\") "
Nov 03 10:52:02.092322 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.092242    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79284696-2fce-4cc6-9224-7e70b7831293-kube-api-access-nszhj\" (UniqueName: \"kubernetes.io/projected/79284696-2fce-4cc6-9224-7e70b7831293-kube-api-access-nszhj\") pod \"79284696-2fce-4cc6-9224-7e70b7831293\" (UID: \"79284696-2fce-4cc6-9224-7e70b7831293\") "
Nov 03 10:52:02.092693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.092663    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79284696-2fce-4cc6-9224-7e70b7831293-plugins-config-volume" pod "79284696-2fce-4cc6-9224-7e70b7831293" (UID: "79284696-2fce-4cc6-9224-7e70b7831293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:02.092866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.092837    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78r5q\" (UniqueName: \"kubernetes.io/projected/d10556d7-e0f7-451d-bba6-752192520de6-kube-api-access-78r5q\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:02.092866 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.092863    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79284696-2fce-4cc6-9224-7e70b7831293-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:02.095113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.095090    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79284696-2fce-4cc6-9224-7e70b7831293-kube-api-access-nszhj" pod "79284696-2fce-4cc6-9224-7e70b7831293" (UID: "79284696-2fce-4cc6-9224-7e70b7831293"). InnerVolumeSpecName "kube-api-access-nszhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:02.194350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.194312    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nszhj\" (UniqueName: \"kubernetes.io/projected/79284696-2fce-4cc6-9224-7e70b7831293-kube-api-access-nszhj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:02.766509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.766048    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"
Nov 03 10:52:02.766509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.766049    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4" event={"ID":"d10556d7-e0f7-451d-bba6-752192520de6","Type":"ContainerDied","Data":"edc61d86f442948bc9d76ccee1bcf11339e43cabc49805bcc87c01d7a02e03cd"}
Nov 03 10:52:02.766509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.766138    5393 scope.go:122] "RemoveContainer" containerID="68fdeb6ff892392bf4f18d58598acc88e6f3fe3d43bfda1d4174add8ff37d742"
Nov 03 10:52:02.769735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.769707    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h" event={"ID":"79284696-2fce-4cc6-9224-7e70b7831293","Type":"ContainerDied","Data":"b349dde1d48bd9d223c3fb67dc6b62b70f44d1a0b3967c02432ff4e2296408ba"}
Nov 03 10:52:02.769881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.769796    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"
Nov 03 10:52:02.781088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.781062    5393 scope.go:122] "RemoveContainer" containerID="31194a66b74bf0f1db063106f217c168cc4d185879c77afc3d8223f024960be9"
Nov 03 10:52:02.789789 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.789734    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"]
Nov 03 10:52:02.793412 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.793387    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-814dc0-5dcdd9d767-ctkf4"]
Nov 03 10:52:02.809837 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.809806    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"]
Nov 03 10:52:02.815620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:02.815595    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-814dc0-9648767d4-t979h"]
Nov 03 10:52:04.230617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:04.230582    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79284696-2fce-4cc6-9224-7e70b7831293" path="/var/lib/kubelet/pods/79284696-2fce-4cc6-9224-7e70b7831293/volumes"
Nov 03 10:52:04.231025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:04.231010    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10556d7-e0f7-451d-bba6-752192520de6" path="/var/lib/kubelet/pods/d10556d7-e0f7-451d-bba6-752192520de6/volumes"
Nov 03 10:52:08.054351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:08.053964    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"]
Nov 03 10:52:08.054351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:08.054323    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy" containerID="cri-o://50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198" gracePeriod=30
Nov 03 10:52:11.136486 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.136444    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"]
Nov 03 10:52:11.137548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.137520    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d10556d7-e0f7-451d-bba6-752192520de6" containerName="payload-pre-processing"
Nov 03 10:52:11.137548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.137549    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10556d7-e0f7-451d-bba6-752192520de6" containerName="payload-pre-processing"
Nov 03 10:52:11.137771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.137585    5393 cpu_manager.go:414] "containerMap: removing container" podUID="79284696-2fce-4cc6-9224-7e70b7831293" containerName="payload-processing"
Nov 03 10:52:11.137771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.137594    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79284696-2fce-4cc6-9224-7e70b7831293" containerName="payload-processing"
Nov 03 10:52:11.137902 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.137794    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10556d7-e0f7-451d-bba6-752192520de6" containerName="payload-pre-processing"
Nov 03 10:52:11.137902 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.137810    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="79284696-2fce-4cc6-9224-7e70b7831293" containerName="payload-processing"
Nov 03 10:52:11.143095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.143071    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.159274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.153405    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-cd6d01-openshift-default-dockercfg-rh2bj\""
Nov 03 10:52:11.159274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.158501    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"]
Nov 03 10:52:11.238507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.238477    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cchn9\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-kube-api-access-cchn9\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.238720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.238529    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-envoy\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.238720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.238559    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-token\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.238720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.238693    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-credential-socket\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.238948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.238836    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-socket\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.238948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.238891    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istiod-ca-cert\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.239056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.238993    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-data\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.239093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.239065    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-podinfo\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.239164 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.239146    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-certs\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.340997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.340956    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-socket\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341033    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istiod-ca-cert\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341075    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-data\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341100    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-podinfo\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341161    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-certs\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341358 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341313    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cchn9\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-kube-api-access-cchn9\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341376    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-envoy\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341382    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-socket\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341405    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-token\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341566 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341435    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-credential-socket\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341566 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341442    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-data\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.341970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.341946    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-certs\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.342115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.342038    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-credential-socket\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.342178 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.342157    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istiod-ca-cert\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.343963 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.343930    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-podinfo\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.344062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.344041    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-envoy\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.380872 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.380833    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cchn9\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-kube-api-access-cchn9\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.381673 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.381618    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-token\") pod \"e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") " pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.472693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.472364    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:11.923728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.923701    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"]
Nov 03 10:52:11.924661 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:52:11.924632    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4f9c29_b268_4c8f_be71_f398b2978ba4.slice/crio-7a80dd80eabb78093aef7972270598a9311c08f0e4997deb2a16753ff427d8b3 WatchSource:0}: Error finding container 7a80dd80eabb78093aef7972270598a9311c08f0e4997deb2a16753ff427d8b3: Status 404 returned error can't find the container with id 7a80dd80eabb78093aef7972270598a9311c08f0e4997deb2a16753ff427d8b3
Nov 03 10:52:11.927410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.927378    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:11.927510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.927458    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:11.927510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:11.927500    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:12.851152 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:12.851117    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" event={"ID":"5d4f9c29-b268-4c8f-be71-f398b2978ba4","Type":"ContainerStarted","Data":"48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f"}
Nov 03 10:52:12.851751 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:12.851168    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" event={"ID":"5d4f9c29-b268-4c8f-be71-f398b2978ba4","Type":"ContainerStarted","Data":"7a80dd80eabb78093aef7972270598a9311c08f0e4997deb2a16753ff427d8b3"}
Nov 03 10:52:12.877669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:12.877618    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podStartSLOduration=1.877602849 podStartE2EDuration="1.877602849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:12.873519059 +0000 UTC m=+1376.778373512" watchObservedRunningTime="2026-11-03 10:52:12.877602849 +0000 UTC m=+1376.782457290"
Nov 03 10:52:13.197149 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:52:13.196643    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:52:13.495821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.495430    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:13.514706 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.506039    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:13.560895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.560862    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:52:13.659537 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.659498    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-envoy\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.659731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.659587    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-data\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.659731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.659649    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-podinfo\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.659731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.659711    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-kube-api-access-pfcm7\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-kube-api-access-pfcm7\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.659889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.659764    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-token\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-token\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.659889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.659851    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-data" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:13.660048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.659894    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istiod-ca-cert\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.660048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.659955    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-credential-socket\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.660150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660084    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-certs\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.660206 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660163    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-socket\") pod \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\" (UID: \"a8c0b587-9ed2-4cbc-922a-74782922dc1a\") "
Nov 03 10:52:13.660309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660287    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istiod-ca-cert" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:13.660486 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660406    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-credential-socket" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:13.660693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660671    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-socket" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:13.660789 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660719    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-certs" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:13.660854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660826    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.660854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660849    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.660960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660865    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.660960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660881    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.660960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.660895    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.662062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.662027    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-envoy" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:13.662209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.662187    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-token" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:13.662464 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.662445    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-kube-api-access-pfcm7" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "kube-api-access-pfcm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:13.662464 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.662454    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-podinfo" pod "a8c0b587-9ed2-4cbc-922a-74782922dc1a" (UID: "a8c0b587-9ed2-4cbc-922a-74782922dc1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:52:13.764387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.764191    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.764387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.764248    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.764387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.764264    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfcm7\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-kube-api-access-pfcm7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.764387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.764280    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c0b587-9ed2-4cbc-922a-74782922dc1a-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:13.859724 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.859692    5393 generic.go:356] "Generic (PLEG): container finished" podID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerID="50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198" exitCode=0
Nov 03 10:52:13.860215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.859744    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"
Nov 03 10:52:13.860215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.859770    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" event={"ID":"a8c0b587-9ed2-4cbc-922a-74782922dc1a","Type":"ContainerDied","Data":"50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198"}
Nov 03 10:52:13.860215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.859826    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf" event={"ID":"a8c0b587-9ed2-4cbc-922a-74782922dc1a","Type":"ContainerDied","Data":"25daac67120701a534e2fbbbf4a1bf2fb9809285d511edac82c7241dd2116633"}
Nov 03 10:52:13.860215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.859858    5393 scope.go:122] "RemoveContainer" containerID="50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198"
Nov 03 10:52:13.860464 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.860409    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:13.861222 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.861206    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:52:13.871309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.871289    5393 scope.go:122] "RemoveContainer" containerID="a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263"
Nov 03 10:52:13.881099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.881073    5393 scope.go:122] "RemoveContainer" containerID="50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198"
Nov 03 10:52:13.881370 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:52:13.881351    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198\": container with ID starting with 50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198 not found: ID does not exist" containerID="50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198"
Nov 03 10:52:13.881443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.881377    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198"} err="failed to get container status \"50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198\": rpc error: code = NotFound desc = could not find container \"50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198\": container with ID starting with 50ad7bd01f25181beb2ec9f444a551934072a19b4a44b32c2dcb87a1bc2be198 not found: ID does not exist"
Nov 03 10:52:13.881443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.881395    5393 scope.go:122] "RemoveContainer" containerID="a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263"
Nov 03 10:52:13.881657 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:52:13.881639    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263\": container with ID starting with a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263 not found: ID does not exist" containerID="a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263"
Nov 03 10:52:13.881740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.881665    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263"} err="failed to get container status \"a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263\": rpc error: code = NotFound desc = could not find container \"a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263\": container with ID starting with a48a967a901960cd473a1b6371408e61032679fed7f09f5e32b0213a0ae5a263 not found: ID does not exist"
Nov 03 10:52:13.905959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.903559    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"]
Nov 03 10:52:13.910229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:13.910199    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-814dc0-openshift-default-64bf5c84df-nkqlf"]
Nov 03 10:52:14.246500 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:14.246049    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" path="/var/lib/kubelet/pods/a8c0b587-9ed2-4cbc-922a-74782922dc1a/volumes"
Nov 03 10:52:16.404242 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.404211    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"]
Nov 03 10:52:16.405007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.404987    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy"
Nov 03 10:52:16.405118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.405009    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy"
Nov 03 10:52:16.405225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.405211    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy"
Nov 03 10:52:16.405281 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.405239    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy"
Nov 03 10:52:16.410169 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.410149    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:16.415873 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.415851    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-cd6d01\""
Nov 03 10:52:16.417488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.417464    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"]
Nov 03 10:52:16.441220 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.441190    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"]
Nov 03 10:52:16.441783 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.441769    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy"
Nov 03 10:52:16.441828 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.441784    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c0b587-9ed2-4cbc-922a-74782922dc1a" containerName="istio-proxy"
Nov 03 10:52:16.445155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.445140    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:16.450944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.450815    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-cd6d01-dockercfg-f7f56\""
Nov 03 10:52:16.456847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.456828    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"]
Nov 03 10:52:16.467273 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.467239    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955a3bdd-2c14-47e9-83fd-31b30d698855-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz\" (UID: \"955a3bdd-2c14-47e9-83fd-31b30d698855\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:16.467477 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.467456    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt544\" (UniqueName: \"kubernetes.io/projected/955a3bdd-2c14-47e9-83fd-31b30d698855-kube-api-access-pt544\") pod \"payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz\" (UID: \"955a3bdd-2c14-47e9-83fd-31b30d698855\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:16.582320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.577462    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt544\" (UniqueName: \"kubernetes.io/projected/955a3bdd-2c14-47e9-83fd-31b30d698855-kube-api-access-pt544\") pod \"payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz\" (UID: \"955a3bdd-2c14-47e9-83fd-31b30d698855\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:16.582320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.577706    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bz6\" (UniqueName: \"kubernetes.io/projected/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-kube-api-access-h9bz6\") pod \"payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6\" (UID: \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:16.582320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.577769    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955a3bdd-2c14-47e9-83fd-31b30d698855-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz\" (UID: \"955a3bdd-2c14-47e9-83fd-31b30d698855\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:16.582320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.577795    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6\" (UID: \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:16.655987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.650239    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955a3bdd-2c14-47e9-83fd-31b30d698855-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz\" (UID: \"955a3bdd-2c14-47e9-83fd-31b30d698855\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:16.655987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.650862    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt544\" (UniqueName: \"kubernetes.io/projected/955a3bdd-2c14-47e9-83fd-31b30d698855-kube-api-access-pt544\") pod \"payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz\" (UID: \"955a3bdd-2c14-47e9-83fd-31b30d698855\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:16.685483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.685115    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bz6\" (UniqueName: \"kubernetes.io/projected/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-kube-api-access-h9bz6\") pod \"payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6\" (UID: \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:16.685776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.685741    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6\" (UID: \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:16.696942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.686791    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6\" (UID: \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:16.730153 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.729304    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bz6\" (UniqueName: \"kubernetes.io/projected/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-kube-api-access-h9bz6\") pod \"payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6\" (UID: \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:16.730153 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.729770    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:16.806942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:16.797805    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:17.126861 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:17.126824    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"]
Nov 03 10:52:17.127795 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:52:17.127770    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955a3bdd_2c14_47e9_83fd_31b30d698855.slice/crio-30bf52acd338ec5b07d961e452fac53bf9a683f418f1f8e8db5c9a2fbb29f82b WatchSource:0}: Error finding container 30bf52acd338ec5b07d961e452fac53bf9a683f418f1f8e8db5c9a2fbb29f82b: Status 404 returned error can't find the container with id 30bf52acd338ec5b07d961e452fac53bf9a683f418f1f8e8db5c9a2fbb29f82b
Nov 03 10:52:17.155387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:17.155362    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"]
Nov 03 10:52:17.156268 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:52:17.156248    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8564fcdb_1c3a_444d_b433_4c04b0fe3e27.slice/crio-45f4dad494a2d773eb447138cd6d97d6256aaa514f14c15b6989ba79bf1d0fe8 WatchSource:0}: Error finding container 45f4dad494a2d773eb447138cd6d97d6256aaa514f14c15b6989ba79bf1d0fe8: Status 404 returned error can't find the container with id 45f4dad494a2d773eb447138cd6d97d6256aaa514f14c15b6989ba79bf1d0fe8
Nov 03 10:52:17.483062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:17.482975    5393 scope.go:122] "RemoveContainer" containerID="5c339e369dce91b4570798d5558560efe06aaf9d8bc1b3700c17b8cef5560bef"
Nov 03 10:52:17.493158 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:17.493137    5393 scope.go:122] "RemoveContainer" containerID="48afe8e6b4159b1fbc2a94a4fb431c6797256dc97aa01dcad0bd62737c00553e"
Nov 03 10:52:17.502726 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:17.502706    5393 scope.go:122] "RemoveContainer" containerID="04ea4b89419d8ee7318a48841f81099e728e46773b02d53086fba49fbf7c8325"
Nov 03 10:52:17.512307 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:17.512287    5393 scope.go:122] "RemoveContainer" containerID="a827cc35e38fe4df4c3c92a33ceee5886deac22d2a28fea404b3250bac24fb31"
Nov 03 10:52:17.525452 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:17.525427    5393 scope.go:122] "RemoveContainer" containerID="b38735d7972402e9cb38b0f7072e767ac25ccad9e3cb880c70ac94109164857d"
Nov 03 10:52:17.535825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:17.535803    5393 scope.go:122] "RemoveContainer" containerID="ffbc4150186c3f50bb061fb51f469214ed1cb9aab3974727b79c4a49acb88143"
Nov 03 10:52:18.030271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:18.029792    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz" event={"ID":"955a3bdd-2c14-47e9-83fd-31b30d698855","Type":"ContainerStarted","Data":"e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06"}
Nov 03 10:52:18.030271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:18.029989    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz" event={"ID":"955a3bdd-2c14-47e9-83fd-31b30d698855","Type":"ContainerStarted","Data":"30bf52acd338ec5b07d961e452fac53bf9a683f418f1f8e8db5c9a2fbb29f82b"}
Nov 03 10:52:18.031627 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:18.031593    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:18.058364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:18.058332    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6" event={"ID":"8564fcdb-1c3a-444d-b433-4c04b0fe3e27","Type":"ContainerStarted","Data":"fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755"}
Nov 03 10:52:18.058364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:18.058388    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6" event={"ID":"8564fcdb-1c3a-444d-b433-4c04b0fe3e27","Type":"ContainerStarted","Data":"45f4dad494a2d773eb447138cd6d97d6256aaa514f14c15b6989ba79bf1d0fe8"}
Nov 03 10:52:18.059353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:18.059311    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:18.070298 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:18.068573    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz" podStartSLOduration=2.068554139 podStartE2EDuration="2.068554139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:18.065955727 +0000 UTC m=+1381.970810166" watchObservedRunningTime="2026-11-03 10:52:18.068554139 +0000 UTC m=+1381.973408581"
Nov 03 10:52:18.085071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:18.085025    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6" podStartSLOduration=2.085011075 podStartE2EDuration="2.085011075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:18.082348463 +0000 UTC m=+1381.987202905" watchObservedRunningTime="2026-11-03 10:52:18.085011075 +0000 UTC m=+1381.989865512"
Nov 03 10:52:23.951585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:23.951556    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"]
Nov 03 10:52:23.959237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:23.959213    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:23.968555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:23.968529    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-08c392\""
Nov 03 10:52:23.968882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:23.968857    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-08c392-dockercfg-6frqm\""
Nov 03 10:52:23.984972 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:23.983062    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"]
Nov 03 10:52:24.053809 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.053772    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6\" (UID: \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:24.054005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.053868    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpnjn\" (UniqueName: \"kubernetes.io/projected/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-kube-api-access-wpnjn\") pod \"payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6\" (UID: \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:24.170447 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.170403    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6\" (UID: \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:24.170740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.170710    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpnjn\" (UniqueName: \"kubernetes.io/projected/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-kube-api-access-wpnjn\") pod \"payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6\" (UID: \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:24.172045 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.172022    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6\" (UID: \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:24.217308 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.215419    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpnjn\" (UniqueName: \"kubernetes.io/projected/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-kube-api-access-wpnjn\") pod \"payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6\" (UID: \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:24.322035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.321993    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"]
Nov 03 10:52:24.323861 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.323832    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:24.692998 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:52:24.692953    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5eaa9d_97f2_4aea_aa45_264fecef58c1.slice/crio-ef97468918dfa4a318f3699b03b48aab86856311e3474a0334cc6f01d8d0ef6f WatchSource:0}: Error finding container ef97468918dfa4a318f3699b03b48aab86856311e3474a0334cc6f01d8d0ef6f: Status 404 returned error can't find the container with id ef97468918dfa4a318f3699b03b48aab86856311e3474a0334cc6f01d8d0ef6f
Nov 03 10:52:24.694659 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:24.694635    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"]
Nov 03 10:52:25.329389 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:25.329356    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6" event={"ID":"5a5eaa9d-97f2-4aea-aa45-264fecef58c1","Type":"ContainerStarted","Data":"0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05"}
Nov 03 10:52:25.329863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:25.329402    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6" event={"ID":"5a5eaa9d-97f2-4aea-aa45-264fecef58c1","Type":"ContainerStarted","Data":"ef97468918dfa4a318f3699b03b48aab86856311e3474a0334cc6f01d8d0ef6f"}
Nov 03 10:52:25.329863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:25.329559    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:25.347488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:25.347423    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6" podStartSLOduration=2.347408257 podStartE2EDuration="2.347408257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:25.347131277 +0000 UTC m=+1389.251985709" watchObservedRunningTime="2026-11-03 10:52:25.347408257 +0000 UTC m=+1389.252262694"
Nov 03 10:52:30.221817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:30.221738    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:52:30.221817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:30.221816    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:52:31.878006 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:31.877970    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"]
Nov 03 10:52:31.894123 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:31.894083    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"]
Nov 03 10:52:36.410662 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:36.410634    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:52:36.836817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:36.836725    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"]
Nov 03 10:52:36.857706 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:36.857674    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"]
Nov 03 10:52:36.857937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:36.857809    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:36.878680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:36.878639    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt" podUID="d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c" containerName="payload-processing" containerID="cri-o://0ff25068211d5839617531409d6f6a9fdbd88be1fd32faab8fa667b92d290e79" gracePeriod=25
Nov 03 10:52:36.884371 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:36.884347    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a35a97-openshift-default-dockercfg-78bjn\""
Nov 03 10:52:36.898298 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:36.897027    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96" podUID="73bdae2d-a392-49a5-85f9-4045f63cf92a" containerName="payload-pre-processing" containerID="cri-o://8a9a13aa5ed1a8c333a5398423d68b46467166fd1562f6799863a93a841c253f" gracePeriod=25
Nov 03 10:52:37.010958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.001014    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-data\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.010958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.001067    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-certs\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.010958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.001112    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istiod-ca-cert\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.010958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.001155    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-envoy\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.010958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.001184    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-token\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.010958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.001245    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-podinfo\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.034952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.032028    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-socket\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.034952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.032173    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xchq4\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-kube-api-access-xchq4\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.034952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.032199    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-credential-socket\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138387    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-data\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138443    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-certs\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138479    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istiod-ca-cert\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138513    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-envoy\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138541    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-token\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138593    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-podinfo\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138625    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-socket\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138715    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xchq4\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-kube-api-access-xchq4\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.138740    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-credential-socket\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.139235    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-credential-socket\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.139497    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-data\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.139820    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-certs\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.163019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.140477    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istiod-ca-cert\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.167021 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.165551    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-socket\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.186944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.173213    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-envoy\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.211755 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.201422    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xchq4\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-kube-api-access-xchq4\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.227698 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.212672    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-podinfo\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.268304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.268235    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-token\") pod \"e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") " pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.483941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.476978    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:37.589653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.589487    5393 generic.go:356] "Generic (PLEG): container finished" podID="d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c" containerID="0ff25068211d5839617531409d6f6a9fdbd88be1fd32faab8fa667b92d290e79" exitCode=0
Nov 03 10:52:37.589653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.589570    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt" event={"ID":"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c","Type":"ContainerDied","Data":"0ff25068211d5839617531409d6f6a9fdbd88be1fd32faab8fa667b92d290e79"}
Nov 03 10:52:37.589911 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.589735    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:52:37.597343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.597307    5393 generic.go:356] "Generic (PLEG): container finished" podID="73bdae2d-a392-49a5-85f9-4045f63cf92a" containerID="8a9a13aa5ed1a8c333a5398423d68b46467166fd1562f6799863a93a841c253f" exitCode=0
Nov 03 10:52:37.597493 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.597413    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96" event={"ID":"73bdae2d-a392-49a5-85f9-4045f63cf92a","Type":"ContainerDied","Data":"8a9a13aa5ed1a8c333a5398423d68b46467166fd1562f6799863a93a841c253f"}
Nov 03 10:52:37.666103 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.666067    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:52:37.758766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.758673    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bdae2d-a392-49a5-85f9-4045f63cf92a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bdae2d-a392-49a5-85f9-4045f63cf92a-plugins-config-volume\") pod \"73bdae2d-a392-49a5-85f9-4045f63cf92a\" (UID: \"73bdae2d-a392-49a5-85f9-4045f63cf92a\") "
Nov 03 10:52:37.758951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.758845    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-plugins-config-volume\") pod \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\" (UID: \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\") "
Nov 03 10:52:37.758951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.758935    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bdae2d-a392-49a5-85f9-4045f63cf92a-kube-api-access-pjqgm\" (UniqueName: \"kubernetes.io/projected/73bdae2d-a392-49a5-85f9-4045f63cf92a-kube-api-access-pjqgm\") pod \"73bdae2d-a392-49a5-85f9-4045f63cf92a\" (UID: \"73bdae2d-a392-49a5-85f9-4045f63cf92a\") "
Nov 03 10:52:37.759113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.759086    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-kube-api-access-wk6zz\" (UniqueName: \"kubernetes.io/projected/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-kube-api-access-wk6zz\") pod \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\" (UID: \"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c\") "
Nov 03 10:52:37.759697 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.759528    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-plugins-config-volume" pod "d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c" (UID: "d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:37.759697 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.759543    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bdae2d-a392-49a5-85f9-4045f63cf92a-plugins-config-volume" pod "73bdae2d-a392-49a5-85f9-4045f63cf92a" (UID: "73bdae2d-a392-49a5-85f9-4045f63cf92a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:37.759862 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.759805    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:37.759862 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.759823    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bdae2d-a392-49a5-85f9-4045f63cf92a-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:37.761761 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.761736    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-kube-api-access-wk6zz" pod "d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c" (UID: "d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c"). InnerVolumeSpecName "kube-api-access-wk6zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:37.761860 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.761778    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bdae2d-a392-49a5-85f9-4045f63cf92a-kube-api-access-pjqgm" pod "73bdae2d-a392-49a5-85f9-4045f63cf92a" (UID: "73bdae2d-a392-49a5-85f9-4045f63cf92a"). InnerVolumeSpecName "kube-api-access-pjqgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:37.834297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.834270    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"]
Nov 03 10:52:37.834983 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:52:37.834952    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8cc142_76a0_4c09_85bb_e5230e83bdbc.slice/crio-77b7bfb1554440495c1544f8fa116924adedfd4e3c281a94611f6f216c8a7925 WatchSource:0}: Error finding container 77b7bfb1554440495c1544f8fa116924adedfd4e3c281a94611f6f216c8a7925: Status 404 returned error can't find the container with id 77b7bfb1554440495c1544f8fa116924adedfd4e3c281a94611f6f216c8a7925
Nov 03 10:52:37.838494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.838456    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:37.838684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.838665    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:37.838820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.838804    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:52:37.887942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.874316    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjqgm\" (UniqueName: \"kubernetes.io/projected/73bdae2d-a392-49a5-85f9-4045f63cf92a-kube-api-access-pjqgm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:37.907035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:37.907003    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6zz\" (UniqueName: \"kubernetes.io/projected/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c-kube-api-access-wk6zz\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:38.606138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.606100    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" event={"ID":"5f8cc142-76a0-4c09-85bb-e5230e83bdbc","Type":"ContainerStarted","Data":"ecc7f2dff91d965b837ae2c8402c05a355bbcd64eeb9722e8922f8eec2876de0"}
Nov 03 10:52:38.606623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.606160    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" event={"ID":"5f8cc142-76a0-4c09-85bb-e5230e83bdbc","Type":"ContainerStarted","Data":"77b7bfb1554440495c1544f8fa116924adedfd4e3c281a94611f6f216c8a7925"}
Nov 03 10:52:38.607610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.607584    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt" event={"ID":"d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c","Type":"ContainerDied","Data":"d4121b00b9d9b0316d0a8935d22886770c66ee44ff59c0ab6d38ac94c5123712"}
Nov 03 10:52:38.607610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.607607    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"
Nov 03 10:52:38.607757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.607644    5393 scope.go:122] "RemoveContainer" containerID="0ff25068211d5839617531409d6f6a9fdbd88be1fd32faab8fa667b92d290e79"
Nov 03 10:52:38.609111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.609090    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"
Nov 03 10:52:38.609221 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.609090    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96" event={"ID":"73bdae2d-a392-49a5-85f9-4045f63cf92a","Type":"ContainerDied","Data":"131cb1faeff89f23fb5b946598b249aec33a5aaa29a44909c60fa27ea2bc8ba3"}
Nov 03 10:52:38.619021 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.618995    5393 scope.go:122] "RemoveContainer" containerID="8a9a13aa5ed1a8c333a5398423d68b46467166fd1562f6799863a93a841c253f"
Nov 03 10:52:38.624863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.624815    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" podStartSLOduration=2.624801995 podStartE2EDuration="2.624801995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:38.623696017 +0000 UTC m=+1402.528550459" watchObservedRunningTime="2026-11-03 10:52:38.624801995 +0000 UTC m=+1402.529656435"
Nov 03 10:52:38.639022 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.638999    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"]
Nov 03 10:52:38.642269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.642247    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b3dc04-647565dcf9-lhk96"]
Nov 03 10:52:38.653807 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.651710    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"]
Nov 03 10:52:38.654824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:38.654806    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b3dc04-668b4fd66c-x9nrt"]
Nov 03 10:52:39.447525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.447485    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc" podUID="c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" containerName="main" containerID="cri-o://61ae8692d82963cacae28678c5bc0d6ccf36695cb5f5c8b6d45e125c9339f482" gracePeriod=45
Nov 03 10:52:39.478180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.478153    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:39.483910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.483888    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:39.619494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.619370    5393 generic.go:356] "Generic (PLEG): container finished" podID="c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" containerID="61ae8692d82963cacae28678c5bc0d6ccf36695cb5f5c8b6d45e125c9339f482" exitCode=0
Nov 03 10:52:39.619765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.619585    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc" event={"ID":"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee","Type":"ContainerDied","Data":"61ae8692d82963cacae28678c5bc0d6ccf36695cb5f5c8b6d45e125c9339f482"}
Nov 03 10:52:39.626072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.623222    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:39.626072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.625510    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:52:39.753548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.753322    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:52:39.839317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.839289    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tls-certs\") pod \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") "
Nov 03 10:52:39.839479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.839349    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-model-cache\") pod \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") "
Nov 03 10:52:39.839479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.839474    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-kube-api-access-kdnws\" (UniqueName: \"kubernetes.io/projected/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-kube-api-access-kdnws\") pod \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") "
Nov 03 10:52:39.839563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.839506    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-dshm\") pod \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") "
Nov 03 10:52:39.839563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.839554    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-home\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-home\") pod \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") "
Nov 03 10:52:39.839675 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.839585    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tmp-dir\") pod \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\" (UID: \"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee\") "
Nov 03 10:52:39.839730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.839660    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-model-cache" pod "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" (UID: "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:39.839895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.839849    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-home" pod "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" (UID: "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:39.840241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.840096    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tmp-dir" pod "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" (UID: "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:39.840524 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.840502    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:39.840594 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.840530    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:39.840594 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.840544    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:39.841854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.841825    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tls-certs" pod "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" (UID: "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:52:39.841955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.841831    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-dshm" pod "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" (UID: "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:39.842176 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.842155    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-kube-api-access-kdnws" pod "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" (UID: "c9c8b89b-bc40-41d3-a0dc-d5715ed62eee"). InnerVolumeSpecName "kube-api-access-kdnws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:39.942370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.942335    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdnws\" (UniqueName: \"kubernetes.io/projected/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-kube-api-access-kdnws\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:39.942370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.942364    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:39.942370 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:39.942373    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:40.249053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:40.249017    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bdae2d-a392-49a5-85f9-4045f63cf92a" path="/var/lib/kubelet/pods/73bdae2d-a392-49a5-85f9-4045f63cf92a/volumes"
Nov 03 10:52:40.249620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:40.249594    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c" path="/var/lib/kubelet/pods/d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c/volumes"
Nov 03 10:52:40.630752 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:40.630722    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"
Nov 03 10:52:40.631255 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:40.630759    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc" event={"ID":"c9c8b89b-bc40-41d3-a0dc-d5715ed62eee","Type":"ContainerDied","Data":"c8f058ffffcf9796005163967fd6daaf9506b1168ee64ba72f343d02960c583f"}
Nov 03 10:52:40.631255 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:40.630813    5393 scope.go:122] "RemoveContainer" containerID="61ae8692d82963cacae28678c5bc0d6ccf36695cb5f5c8b6d45e125c9339f482"
Nov 03 10:52:40.661727 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:40.651655    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"]
Nov 03 10:52:40.661727 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:40.654056    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a51d55/ipp-route-a51d55-kserve-7745757bbf-nlgvc"]
Nov 03 10:52:41.118853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:41.118815    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"]
Nov 03 10:52:41.119241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:41.119206    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" containerID="cri-o://0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922" gracePeriod=30
Nov 03 10:52:41.121977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:41.121955    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"]
Nov 03 10:52:41.122191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:41.122169    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy" containerID="cri-o://92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279" gracePeriod=30
Nov 03 10:52:42.272482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.272442    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" path="/var/lib/kubelet/pods/c9c8b89b-bc40-41d3-a0dc-d5715ed62eee/volumes"
Nov 03 10:52:42.325187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325158    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"]
Nov 03 10:52:42.325731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325718    5393 cpu_manager.go:414] "containerMap: removing container" podUID="73bdae2d-a392-49a5-85f9-4045f63cf92a" containerName="payload-pre-processing"
Nov 03 10:52:42.325790 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325732    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bdae2d-a392-49a5-85f9-4045f63cf92a" containerName="payload-pre-processing"
Nov 03 10:52:42.325790 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325754    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" containerName="main"
Nov 03 10:52:42.325790 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325759    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" containerName="main"
Nov 03 10:52:42.325889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325799    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c" containerName="payload-processing"
Nov 03 10:52:42.325889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325805    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c" containerName="payload-processing"
Nov 03 10:52:42.325966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325897    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c8b89b-bc40-41d3-a0dc-d5715ed62eee" containerName="main"
Nov 03 10:52:42.325966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325911    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ad447e-2b01-4e88-9e3d-0e0c61a1e57c" containerName="payload-processing"
Nov 03 10:52:42.325966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.325941    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bdae2d-a392-49a5-85f9-4045f63cf92a" containerName="payload-pre-processing"
Nov 03 10:52:42.330696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.330678    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:42.336776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.336745    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a35a97-dockercfg-hb9qv\""
Nov 03 10:52:42.336931 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.336796    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a35a97\""
Nov 03 10:52:42.341468 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.341443    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"]
Nov 03 10:52:42.345776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.345732    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vhc\" (UniqueName: \"kubernetes.io/projected/92f6573b-c38d-430f-9634-6fd8823cad2e-kube-api-access-m4vhc\") pod \"payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx\" (UID: \"92f6573b-c38d-430f-9634-6fd8823cad2e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:42.345965 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.345889    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f6573b-c38d-430f-9634-6fd8823cad2e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx\" (UID: \"92f6573b-c38d-430f-9634-6fd8823cad2e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:42.357067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.357038    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"]
Nov 03 10:52:42.361167 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.361148    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:42.370149 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.370122    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"]
Nov 03 10:52:42.518952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.516853    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vhc\" (UniqueName: \"kubernetes.io/projected/92f6573b-c38d-430f-9634-6fd8823cad2e-kube-api-access-m4vhc\") pod \"payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx\" (UID: \"92f6573b-c38d-430f-9634-6fd8823cad2e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:42.518952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.517092    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f6573b-c38d-430f-9634-6fd8823cad2e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx\" (UID: \"92f6573b-c38d-430f-9634-6fd8823cad2e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:42.580996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.555415    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f6573b-c38d-430f-9634-6fd8823cad2e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx\" (UID: \"92f6573b-c38d-430f-9634-6fd8823cad2e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:42.580996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.566655    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vhc\" (UniqueName: \"kubernetes.io/projected/92f6573b-c38d-430f-9634-6fd8823cad2e-kube-api-access-m4vhc\") pod \"payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx\" (UID: \"92f6573b-c38d-430f-9634-6fd8823cad2e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:42.620947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.619977    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgrvn\" (UniqueName: \"kubernetes.io/projected/b405dc6a-ff68-46db-a6a5-21b93e6c094d-kube-api-access-vgrvn\") pod \"payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8\" (UID: \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:42.620947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.620198    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b405dc6a-ff68-46db-a6a5-21b93e6c094d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8\" (UID: \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:42.667950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.661530    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:42.721614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.721578    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrvn\" (UniqueName: \"kubernetes.io/projected/b405dc6a-ff68-46db-a6a5-21b93e6c094d-kube-api-access-vgrvn\") pod \"payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8\" (UID: \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:42.721802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.721786    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b405dc6a-ff68-46db-a6a5-21b93e6c094d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8\" (UID: \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:42.722814 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.722781    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b405dc6a-ff68-46db-a6a5-21b93e6c094d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8\" (UID: \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:42.767941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.763278    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrvn\" (UniqueName: \"kubernetes.io/projected/b405dc6a-ff68-46db-a6a5-21b93e6c094d-kube-api-access-vgrvn\") pod \"payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8\" (UID: \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:42.997176 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:42.985290    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:43.132934 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.132884    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"]
Nov 03 10:52:43.209111 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:52:43.209067    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:52:43.339067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.339032    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"]
Nov 03 10:52:43.775721 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.775632    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8" event={"ID":"b405dc6a-ff68-46db-a6a5-21b93e6c094d","Type":"ContainerStarted","Data":"fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8"}
Nov 03 10:52:43.776008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.775984    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8" event={"ID":"b405dc6a-ff68-46db-a6a5-21b93e6c094d","Type":"ContainerStarted","Data":"62ee253423aafe6bdf552ef496b35efe22679d316c528f6776f3fcc895db0f9f"}
Nov 03 10:52:43.777364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.777267    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:43.779951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.779892    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx" event={"ID":"92f6573b-c38d-430f-9634-6fd8823cad2e","Type":"ContainerStarted","Data":"d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00"}
Nov 03 10:52:43.780262 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.780232    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx" event={"ID":"92f6573b-c38d-430f-9634-6fd8823cad2e","Type":"ContainerStarted","Data":"00e14136a8de21c46d4b00ffac59b91bb3bbb6ee42ca7eb78c929f5f5270ea3d"}
Nov 03 10:52:43.782020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.782001    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:43.823112 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.822441    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8" podStartSLOduration=1.822420583 podStartE2EDuration="1.822420583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:43.818275373 +0000 UTC m=+1407.723129815" watchObservedRunningTime="2026-11-03 10:52:43.822420583 +0000 UTC m=+1407.727275027"
Nov 03 10:52:43.841941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:43.840893    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx" podStartSLOduration=1.840874772 podStartE2EDuration="1.840874772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:43.8405878 +0000 UTC m=+1407.745442242" watchObservedRunningTime="2026-11-03 10:52:43.840874772 +0000 UTC m=+1407.745729214"
Nov 03 10:52:46.187238 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.187203    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 10:52:46.187690 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.187268    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 10:52:46.416648 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.416618    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:52:46.448368 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.448341    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:52:46.532037 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.531997    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-istio-token\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-istio-token\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.532224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532050    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-podinfo\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.532224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532093    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-credential-socket\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.532224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532120    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-kube-api-access-2c6c4\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-kube-api-access-2c6c4\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.532224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532152    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-socket\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.532224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532194    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-certs\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532230    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-envoy\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532294    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-credential-socket\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532327    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f79b86c-769b-47aa-a876-adc74caf37ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f79b86c-769b-47aa-a876-adc74caf37ef-istiod-ca-cert\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532346    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-credential-socket" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532371    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-data\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532412    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-envoy\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532471    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-socket" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532487    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea688a05-d023-4385-8f9c-54ceac47a320-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea688a05-d023-4385-8f9c-54ceac47a320-istiod-ca-cert\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.532504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532486    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-certs" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532528    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-kube-api-access-r7qm7\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-kube-api-access-r7qm7\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532561    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-credential-socket" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532581    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-socket\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532677    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea688a05-d023-4385-8f9c-54ceac47a320-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea688a05-d023-4385-8f9c-54ceac47a320-istio-podinfo\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532689    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-data" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532726    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-certs\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532771    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-data\") pod \"ea688a05-d023-4385-8f9c-54ceac47a320\" (UID: \"ea688a05-d023-4385-8f9c-54ceac47a320\") "
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532810    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-token\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-token\") pod \"5f79b86c-769b-47aa-a876-adc74caf37ef\" (UID: \"5f79b86c-769b-47aa-a876-adc74caf37ef\") "
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532950    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea688a05-d023-4385-8f9c-54ceac47a320-istiod-ca-cert" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:46.533174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.532995    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f79b86c-769b-47aa-a876-adc74caf37ef-istiod-ca-cert" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:46.534201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.533718    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.534201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.533745    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.534201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.533761    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.534201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.533785    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f79b86c-769b-47aa-a876-adc74caf37ef-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.534201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.533801    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.534201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.533814    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea688a05-d023-4385-8f9c-54ceac47a320-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.534201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.533828    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.534201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.533937    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-certs" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.534998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.534967    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-podinfo" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:52:46.535122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.535002    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-istio-token" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:46.535226 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.535201    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-data" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.535345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.535235    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-socket" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.535479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.535452    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-kube-api-access-2c6c4" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "kube-api-access-2c6c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:46.535629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.535606    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea688a05-d023-4385-8f9c-54ceac47a320-istio-podinfo" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:52:46.535629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.535610    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-envoy" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.535747 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.535642    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-envoy" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:52:46.536085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.536070    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-token" pod "5f79b86c-769b-47aa-a876-adc74caf37ef" (UID: "5f79b86c-769b-47aa-a876-adc74caf37ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:46.536482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.536468    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-kube-api-access-r7qm7" pod "ea688a05-d023-4385-8f9c-54ceac47a320" (UID: "ea688a05-d023-4385-8f9c-54ceac47a320"). InnerVolumeSpecName "kube-api-access-r7qm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635068    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635101    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7qm7\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-kube-api-access-r7qm7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635118    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635133    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea688a05-d023-4385-8f9c-54ceac47a320-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635148    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f79b86c-769b-47aa-a876-adc74caf37ef-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635161    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635175    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635189    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea688a05-d023-4385-8f9c-54ceac47a320-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635202    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f79b86c-769b-47aa-a876-adc74caf37ef-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635216    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c6c4\" (UniqueName: \"kubernetes.io/projected/5f79b86c-769b-47aa-a876-adc74caf37ef-kube-api-access-2c6c4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.635264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.635230    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea688a05-d023-4385-8f9c-54ceac47a320-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.638679    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"]
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.639453    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy"
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.639468    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy"
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.639498    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy"
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.639503    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy"
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.639528    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy"
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.639532    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy"
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.639623    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy"
Nov 03 10:52:46.640538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.639639    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerName="istio-proxy"
Nov 03 10:52:46.672556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.671150    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"]
Nov 03 10:52:46.672556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.671286    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.681669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.679165    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:52:46.681669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.679932    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:52:46.755685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.755622    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnbwh\" (UniqueName: \"kubernetes.io/projected/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-kube-api-access-pnbwh\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.755685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.755674    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.757234 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.755771    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.842372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.842340    5393 generic.go:356] "Generic (PLEG): container finished" podID="ea688a05-d023-4385-8f9c-54ceac47a320" containerID="0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922" exitCode=0
Nov 03 10:52:46.842539 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.842392    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"
Nov 03 10:52:46.842539 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.842423    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" event={"ID":"ea688a05-d023-4385-8f9c-54ceac47a320","Type":"ContainerDied","Data":"0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922"}
Nov 03 10:52:46.842539 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.842478    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm" event={"ID":"ea688a05-d023-4385-8f9c-54ceac47a320","Type":"ContainerDied","Data":"f81532d6c3f955f79dca3a31d02241371685caf5798ae58c142e315c895b5bcd"}
Nov 03 10:52:46.842539 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.842501    5393 scope.go:122] "RemoveContainer" containerID="0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922"
Nov 03 10:52:46.844077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.844003    5393 generic.go:356] "Generic (PLEG): container finished" podID="5f79b86c-769b-47aa-a876-adc74caf37ef" containerID="92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279" exitCode=0
Nov 03 10:52:46.844077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.844053    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"
Nov 03 10:52:46.844474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.844050    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" event={"ID":"5f79b86c-769b-47aa-a876-adc74caf37ef","Type":"ContainerDied","Data":"92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279"}
Nov 03 10:52:46.844474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.844196    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx" event={"ID":"5f79b86c-769b-47aa-a876-adc74caf37ef","Type":"ContainerDied","Data":"35d7b3cbecc2581c41c6c4b8b57050f0909922e802e77e18010c8b740f0e5d42"}
Nov 03 10:52:46.860538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.860505    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnbwh\" (UniqueName: \"kubernetes.io/projected/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-kube-api-access-pnbwh\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.860860 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.860838    5393 scope.go:122] "RemoveContainer" containerID="92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8"
Nov 03 10:52:46.861074 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.860853    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.862994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.861633    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.862994 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.862570    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.863790 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.863763    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.868195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.868176    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnbwh\" (UniqueName: \"kubernetes.io/projected/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-kube-api-access-pnbwh\") pod \"maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:46.874480 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.874461    5393 scope.go:122] "RemoveContainer" containerID="0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922"
Nov 03 10:52:46.874996 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:52:46.874967    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922\": container with ID starting with 0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922 not found: ID does not exist" containerID="0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922"
Nov 03 10:52:46.875097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.875005    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922"} err="failed to get container status \"0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922\": rpc error: code = NotFound desc = could not find container \"0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922\": container with ID starting with 0f3845052bd8f842a5199d753efa2570921b08b71abc2a469bfb6839b076b922 not found: ID does not exist"
Nov 03 10:52:46.875097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.875023    5393 scope.go:122] "RemoveContainer" containerID="92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8"
Nov 03 10:52:46.875296 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:52:46.875269    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8\": container with ID starting with 92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8 not found: ID does not exist" containerID="92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8"
Nov 03 10:52:46.875408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.875301    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8"} err="failed to get container status \"92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8\": rpc error: code = NotFound desc = could not find container \"92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8\": container with ID starting with 92f6215921acdd26040ff7e0a1ebe70d46d38ce12668010310ab41c580cbbae8 not found: ID does not exist"
Nov 03 10:52:46.875408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.875319    5393 scope.go:122] "RemoveContainer" containerID="92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279"
Nov 03 10:52:46.876413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.876396    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"]
Nov 03 10:52:46.879953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.879931    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-jmnfx"]
Nov 03 10:52:46.885404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.885384    5393 scope.go:122] "RemoveContainer" containerID="92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279"
Nov 03 10:52:46.885641 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:52:46.885621    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279\": container with ID starting with 92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279 not found: ID does not exist" containerID="92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279"
Nov 03 10:52:46.885686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.885648    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279"} err="failed to get container status \"92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279\": rpc error: code = NotFound desc = could not find container \"92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279\": container with ID starting with 92971bdce591c89acef32fbb071ae133e1581e0aa3f7c0d85d681a31d7afe279 not found: ID does not exist"
Nov 03 10:52:46.891426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.891346    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"]
Nov 03 10:52:46.894471 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:46.894454    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b3dc04-openshift-default-78ddc9b56f-f4jlm"]
Nov 03 10:52:47.005432 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:47.005401    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:47.291161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:47.291131    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"]
Nov 03 10:52:47.292739 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:52:47.292705    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fadd0de_215f_4ac0_9505_0bc2980bc3eb.slice/crio-b869c1df17fc6fbeba7eda3427662d8aede3027602ba718f0e4787ac1f5809ea WatchSource:0}: Error finding container b869c1df17fc6fbeba7eda3427662d8aede3027602ba718f0e4787ac1f5809ea: Status 404 returned error can't find the container with id b869c1df17fc6fbeba7eda3427662d8aede3027602ba718f0e4787ac1f5809ea
Nov 03 10:52:47.855908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:47.855872    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr" event={"ID":"2fadd0de-215f-4ac0-9505-0bc2980bc3eb","Type":"ContainerStarted","Data":"f97076ad29cb5d625274bd53d9e37c979c6ba3fa09d62e927c5db312c0059e60"}
Nov 03 10:52:47.856117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:47.855945    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr" event={"ID":"2fadd0de-215f-4ac0-9505-0bc2980bc3eb","Type":"ContainerStarted","Data":"b869c1df17fc6fbeba7eda3427662d8aede3027602ba718f0e4787ac1f5809ea"}
Nov 03 10:52:47.873987 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:47.873934    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr" podStartSLOduration=1.873905277 podStartE2EDuration="1.873905277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:52:47.870184724 +0000 UTC m=+1411.775039165" watchObservedRunningTime="2026-11-03 10:52:47.873905277 +0000 UTC m=+1411.778759737"
Nov 03 10:52:48.231443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:48.231366    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f79b86c-769b-47aa-a876-adc74caf37ef" path="/var/lib/kubelet/pods/5f79b86c-769b-47aa-a876-adc74caf37ef/volumes"
Nov 03 10:52:48.231881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:48.231864    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" path="/var/lib/kubelet/pods/ea688a05-d023-4385-8f9c-54ceac47a320/volumes"
Nov 03 10:52:48.864974 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:48.864946    5393 generic.go:356] "Generic (PLEG): container finished" podID="2fadd0de-215f-4ac0-9505-0bc2980bc3eb" containerID="f97076ad29cb5d625274bd53d9e37c979c6ba3fa09d62e927c5db312c0059e60" exitCode=0
Nov 03 10:52:48.865375 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:48.865021    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr" event={"ID":"2fadd0de-215f-4ac0-9505-0bc2980bc3eb","Type":"ContainerDied","Data":"f97076ad29cb5d625274bd53d9e37c979c6ba3fa09d62e927c5db312c0059e60"}
Nov 03 10:52:50.142995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.141495    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:50.236521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.236479    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-openshift-ca\") pod \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") "
Nov 03 10:52:50.236720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.236551    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-kube-api-access-pnbwh\" (UniqueName: \"kubernetes.io/projected/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-kube-api-access-pnbwh\") pod \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") "
Nov 03 10:52:50.236720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.236637    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-xks-ca\") pod \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\" (UID: \"2fadd0de-215f-4ac0-9505-0bc2980bc3eb\") "
Nov 03 10:52:50.238108 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.238074    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-openshift-ca" pod "2fadd0de-215f-4ac0-9505-0bc2980bc3eb" (UID: "2fadd0de-215f-4ac0-9505-0bc2980bc3eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:52:50.240536 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.240505    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:50.286619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.270147    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-kube-api-access-pnbwh" pod "2fadd0de-215f-4ac0-9505-0bc2980bc3eb" (UID: "2fadd0de-215f-4ac0-9505-0bc2980bc3eb"). InnerVolumeSpecName "kube-api-access-pnbwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:52:50.286619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.275632    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-xks-ca" pod "2fadd0de-215f-4ac0-9505-0bc2980bc3eb" (UID: "2fadd0de-215f-4ac0-9505-0bc2980bc3eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:52:50.342164 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.342135    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnbwh\" (UniqueName: \"kubernetes.io/projected/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-kube-api-access-pnbwh\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:50.342164 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.342163    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fadd0de-215f-4ac0-9505-0bc2980bc3eb-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:52:50.912952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.909433    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr" event={"ID":"2fadd0de-215f-4ac0-9505-0bc2980bc3eb","Type":"ContainerDied","Data":"b869c1df17fc6fbeba7eda3427662d8aede3027602ba718f0e4787ac1f5809ea"}
Nov 03 10:52:50.912952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.909533    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b869c1df17fc6fbeba7eda3427662d8aede3027602ba718f0e4787ac1f5809ea"
Nov 03 10:52:50.912952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:50.909603    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"
Nov 03 10:52:54.684310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:54.684272    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"]
Nov 03 10:52:54.693837 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:54.693807    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"]
Nov 03 10:52:55.837630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:55.837599    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:52:55.838145 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:55.837702    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:52:56.758653 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:56.758616    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"]
Nov 03 10:52:56.769677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:56.769647    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"]
Nov 03 10:52:59.693939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:59.693336    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p" podUID="5afc2511-2821-4359-a513-63a2f2aaa5b6" containerName="payload-processing" containerID="cri-o://53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21" gracePeriod=25
Nov 03 10:52:59.704955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:52:59.700579    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc" podUID="829f3652-eb44-4271-acf2-076be8e109bf" containerName="payload-pre-processing" containerID="cri-o://e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189" gracePeriod=25
Nov 03 10:53:00.025369 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.024975    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:53:00.042614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.042318    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:53:00.137172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.137138    5393 generic.go:356] "Generic (PLEG): container finished" podID="5afc2511-2821-4359-a513-63a2f2aaa5b6" containerID="53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21" exitCode=0
Nov 03 10:53:00.137330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.137190    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"
Nov 03 10:53:00.137330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.137213    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p" event={"ID":"5afc2511-2821-4359-a513-63a2f2aaa5b6","Type":"ContainerDied","Data":"53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21"}
Nov 03 10:53:00.137330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.137295    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p" event={"ID":"5afc2511-2821-4359-a513-63a2f2aaa5b6","Type":"ContainerDied","Data":"a6f701de826808e43a434bf5ff573129bb28a44fde6c03811f5917fd747b10fb"}
Nov 03 10:53:00.137330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.137321    5393 scope.go:122] "RemoveContainer" containerID="53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21"
Nov 03 10:53:00.138626 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.138604    5393 generic.go:356] "Generic (PLEG): container finished" podID="829f3652-eb44-4271-acf2-076be8e109bf" containerID="e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189" exitCode=0
Nov 03 10:53:00.138689 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.138641    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"
Nov 03 10:53:00.138728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.138692    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc" event={"ID":"829f3652-eb44-4271-acf2-076be8e109bf","Type":"ContainerDied","Data":"e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189"}
Nov 03 10:53:00.138770 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.138728    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc" event={"ID":"829f3652-eb44-4271-acf2-076be8e109bf","Type":"ContainerDied","Data":"66b040cac5cd50887755da9b24e10e0ad56c9bd0454ec13753fedd6e2b75f31c"}
Nov 03 10:53:00.148185 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.148161    5393 scope.go:122] "RemoveContainer" containerID="53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21"
Nov 03 10:53:00.148483 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:00.148461    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21\": container with ID starting with 53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21 not found: ID does not exist" containerID="53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21"
Nov 03 10:53:00.148576 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.148490    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21"} err="failed to get container status \"53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21\": rpc error: code = NotFound desc = could not find container \"53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21\": container with ID starting with 53308662a0c5b52386836d6b9e89c1fa1b6679e37c11ee5dd293f6aecb968f21 not found: ID does not exist"
Nov 03 10:53:00.148576 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.148509    5393 scope.go:122] "RemoveContainer" containerID="e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189"
Nov 03 10:53:00.158191 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.158172    5393 scope.go:122] "RemoveContainer" containerID="e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189"
Nov 03 10:53:00.158503 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:00.158483    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189\": container with ID starting with e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189 not found: ID does not exist" containerID="e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189"
Nov 03 10:53:00.158568 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.158511    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189"} err="failed to get container status \"e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189\": rpc error: code = NotFound desc = could not find container \"e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189\": container with ID starting with e904e59e55cb0cedc9e3fdd95a8e9fc24b7bb0414cf1ec4f20dc3b90b8cd9189 not found: ID does not exist"
Nov 03 10:53:00.180008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.179931    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829f3652-eb44-4271-acf2-076be8e109bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f3652-eb44-4271-acf2-076be8e109bf-plugins-config-volume\") pod \"829f3652-eb44-4271-acf2-076be8e109bf\" (UID: \"829f3652-eb44-4271-acf2-076be8e109bf\") "
Nov 03 10:53:00.180214 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.179978    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5afc2511-2821-4359-a513-63a2f2aaa5b6-kube-api-access-ttp6c\" (UniqueName: \"kubernetes.io/projected/5afc2511-2821-4359-a513-63a2f2aaa5b6-kube-api-access-ttp6c\") pod \"5afc2511-2821-4359-a513-63a2f2aaa5b6\" (UID: \"5afc2511-2821-4359-a513-63a2f2aaa5b6\") "
Nov 03 10:53:00.180373 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.180343    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829f3652-eb44-4271-acf2-076be8e109bf-plugins-config-volume" pod "829f3652-eb44-4271-acf2-076be8e109bf" (UID: "829f3652-eb44-4271-acf2-076be8e109bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:00.180479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.180396    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829f3652-eb44-4271-acf2-076be8e109bf-kube-api-access-gbtbh\" (UniqueName: \"kubernetes.io/projected/829f3652-eb44-4271-acf2-076be8e109bf-kube-api-access-gbtbh\") pod \"829f3652-eb44-4271-acf2-076be8e109bf\" (UID: \"829f3652-eb44-4271-acf2-076be8e109bf\") "
Nov 03 10:53:00.180517 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.180502    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5afc2511-2821-4359-a513-63a2f2aaa5b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc2511-2821-4359-a513-63a2f2aaa5b6-plugins-config-volume\") pod \"5afc2511-2821-4359-a513-63a2f2aaa5b6\" (UID: \"5afc2511-2821-4359-a513-63a2f2aaa5b6\") "
Nov 03 10:53:00.180853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.180831    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afc2511-2821-4359-a513-63a2f2aaa5b6-plugins-config-volume" pod "5afc2511-2821-4359-a513-63a2f2aaa5b6" (UID: "5afc2511-2821-4359-a513-63a2f2aaa5b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:00.181072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.181057    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc2511-2821-4359-a513-63a2f2aaa5b6-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:00.181115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.181074    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f3652-eb44-4271-acf2-076be8e109bf-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:00.182409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.182393    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afc2511-2821-4359-a513-63a2f2aaa5b6-kube-api-access-ttp6c" pod "5afc2511-2821-4359-a513-63a2f2aaa5b6" (UID: "5afc2511-2821-4359-a513-63a2f2aaa5b6"). InnerVolumeSpecName "kube-api-access-ttp6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:00.182601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.182572    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829f3652-eb44-4271-acf2-076be8e109bf-kube-api-access-gbtbh" pod "829f3652-eb44-4271-acf2-076be8e109bf" (UID: "829f3652-eb44-4271-acf2-076be8e109bf"). InnerVolumeSpecName "kube-api-access-gbtbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:00.285942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.285782    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttp6c\" (UniqueName: \"kubernetes.io/projected/5afc2511-2821-4359-a513-63a2f2aaa5b6-kube-api-access-ttp6c\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:00.285942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.285858    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbtbh\" (UniqueName: \"kubernetes.io/projected/829f3652-eb44-4271-acf2-076be8e109bf-kube-api-access-gbtbh\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:00.459700 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.459662    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"]
Nov 03 10:53:00.462705 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.462683    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-835861-7dcf7b75f7-knv9p"]
Nov 03 10:53:00.471055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.471030    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"]
Nov 03 10:53:00.474290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:00.474267    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-835861-f6f9b8767-pz4xc"]
Nov 03 10:53:01.759044 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:01.759003    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c" podUID="c01721d7-ef7a-40d9-af85-62b1bfa78f97" containerName="payload-processing" containerID="cri-o://c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b" gracePeriod=25
Nov 03 10:53:01.770069 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:01.770026    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr" podUID="58bad0a0-d5f1-469d-b11a-26af1f55fe16" containerName="payload-pre-processing" containerID="cri-o://8b3024034c9b419a9bd1ea7f570fcd5802c162855635ba426b90b096f504522f" gracePeriod=25
Nov 03 10:53:02.161356 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.161329    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:53:02.164025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.163980    5393 generic.go:356] "Generic (PLEG): container finished" podID="c01721d7-ef7a-40d9-af85-62b1bfa78f97" containerID="c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b" exitCode=0
Nov 03 10:53:02.164189 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.164078    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"
Nov 03 10:53:02.164265 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.164210    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c" event={"ID":"c01721d7-ef7a-40d9-af85-62b1bfa78f97","Type":"ContainerDied","Data":"c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b"}
Nov 03 10:53:02.164328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.164264    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c" event={"ID":"c01721d7-ef7a-40d9-af85-62b1bfa78f97","Type":"ContainerDied","Data":"c12a86764ff652defdc4b5529764ca46d25acc98c139505ef0979c3111a1f373"}
Nov 03 10:53:02.164328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.164296    5393 scope.go:122] "RemoveContainer" containerID="c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b"
Nov 03 10:53:02.166545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.166517    5393 generic.go:356] "Generic (PLEG): container finished" podID="58bad0a0-d5f1-469d-b11a-26af1f55fe16" containerID="8b3024034c9b419a9bd1ea7f570fcd5802c162855635ba426b90b096f504522f" exitCode=0
Nov 03 10:53:02.166680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.166558    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr" event={"ID":"58bad0a0-d5f1-469d-b11a-26af1f55fe16","Type":"ContainerDied","Data":"8b3024034c9b419a9bd1ea7f570fcd5802c162855635ba426b90b096f504522f"}
Nov 03 10:53:02.175524 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.175498    5393 scope.go:122] "RemoveContainer" containerID="c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b"
Nov 03 10:53:02.175846 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:02.175823    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b\": container with ID starting with c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b not found: ID does not exist" containerID="c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b"
Nov 03 10:53:02.175974 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.175856    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b"} err="failed to get container status \"c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b\": rpc error: code = NotFound desc = could not find container \"c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b\": container with ID starting with c60c9d191246631de33b212e379c9b0fbf6f7c905a3bcd0cb256edec74bae15b not found: ID does not exist"
Nov 03 10:53:02.204084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.204056    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:53:02.237513 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.237482    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5afc2511-2821-4359-a513-63a2f2aaa5b6" path="/var/lib/kubelet/pods/5afc2511-2821-4359-a513-63a2f2aaa5b6/volumes"
Nov 03 10:53:02.238448 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.238029    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829f3652-eb44-4271-acf2-076be8e109bf" path="/var/lib/kubelet/pods/829f3652-eb44-4271-acf2-076be8e109bf/volumes"
Nov 03 10:53:02.331174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.331086    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bad0a0-d5f1-469d-b11a-26af1f55fe16-kube-api-access-46zvm\" (UniqueName: \"kubernetes.io/projected/58bad0a0-d5f1-469d-b11a-26af1f55fe16-kube-api-access-46zvm\") pod \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\" (UID: \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\") "
Nov 03 10:53:02.331174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.331152    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01721d7-ef7a-40d9-af85-62b1bfa78f97-kube-api-access-99kss\" (UniqueName: \"kubernetes.io/projected/c01721d7-ef7a-40d9-af85-62b1bfa78f97-kube-api-access-99kss\") pod \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\" (UID: \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\") "
Nov 03 10:53:02.331413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.331210    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01721d7-ef7a-40d9-af85-62b1bfa78f97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01721d7-ef7a-40d9-af85-62b1bfa78f97-plugins-config-volume\") pod \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\" (UID: \"c01721d7-ef7a-40d9-af85-62b1bfa78f97\") "
Nov 03 10:53:02.331413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.331244    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58bad0a0-d5f1-469d-b11a-26af1f55fe16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bad0a0-d5f1-469d-b11a-26af1f55fe16-plugins-config-volume\") pod \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\" (UID: \"58bad0a0-d5f1-469d-b11a-26af1f55fe16\") "
Nov 03 10:53:02.331690 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.331664    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01721d7-ef7a-40d9-af85-62b1bfa78f97-plugins-config-volume" pod "c01721d7-ef7a-40d9-af85-62b1bfa78f97" (UID: "c01721d7-ef7a-40d9-af85-62b1bfa78f97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:02.331757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.331665    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bad0a0-d5f1-469d-b11a-26af1f55fe16-plugins-config-volume" pod "58bad0a0-d5f1-469d-b11a-26af1f55fe16" (UID: "58bad0a0-d5f1-469d-b11a-26af1f55fe16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:02.331823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.331800    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01721d7-ef7a-40d9-af85-62b1bfa78f97-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:02.331823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.331813    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bad0a0-d5f1-469d-b11a-26af1f55fe16-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:02.333572 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.333544    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01721d7-ef7a-40d9-af85-62b1bfa78f97-kube-api-access-99kss" pod "c01721d7-ef7a-40d9-af85-62b1bfa78f97" (UID: "c01721d7-ef7a-40d9-af85-62b1bfa78f97"). InnerVolumeSpecName "kube-api-access-99kss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:02.333572 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.333567    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bad0a0-d5f1-469d-b11a-26af1f55fe16-kube-api-access-46zvm" pod "58bad0a0-d5f1-469d-b11a-26af1f55fe16" (UID: "58bad0a0-d5f1-469d-b11a-26af1f55fe16"). InnerVolumeSpecName "kube-api-access-46zvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:02.433120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.433085    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zvm\" (UniqueName: \"kubernetes.io/projected/58bad0a0-d5f1-469d-b11a-26af1f55fe16-kube-api-access-46zvm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:02.433120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.433117    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99kss\" (UniqueName: \"kubernetes.io/projected/c01721d7-ef7a-40d9-af85-62b1bfa78f97-kube-api-access-99kss\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:02.491989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.491905    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"]
Nov 03 10:53:02.494824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:02.494797    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f81a79-57487c7bf7-wb66c"]
Nov 03 10:53:03.174294 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:03.174261    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"
Nov 03 10:53:03.174776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:03.174335    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr" event={"ID":"58bad0a0-d5f1-469d-b11a-26af1f55fe16","Type":"ContainerDied","Data":"eaa1b6b5ba3dc8c277d06c3e238cd946ad351a4b53ce38bd7db6627900fb3843"}
Nov 03 10:53:03.174776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:03.174429    5393 scope.go:122] "RemoveContainer" containerID="8b3024034c9b419a9bd1ea7f570fcd5802c162855635ba426b90b096f504522f"
Nov 03 10:53:03.215587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:03.215548    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"]
Nov 03 10:53:03.221385 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:03.221360    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f81a79-868ddc955-mwscr"]
Nov 03 10:53:04.231961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:04.231910    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58bad0a0-d5f1-469d-b11a-26af1f55fe16" path="/var/lib/kubelet/pods/58bad0a0-d5f1-469d-b11a-26af1f55fe16/volumes"
Nov 03 10:53:04.232484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:04.232467    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01721d7-ef7a-40d9-af85-62b1bfa78f97" path="/var/lib/kubelet/pods/c01721d7-ef7a-40d9-af85-62b1bfa78f97/volumes"
Nov 03 10:53:05.519307 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:05.519268    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"]
Nov 03 10:53:05.522187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:05.522164    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f81a79-h7qfr"]
Nov 03 10:53:06.230698 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:06.230665    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fadd0de-215f-4ac0-9505-0bc2980bc3eb" path="/var/lib/kubelet/pods/2fadd0de-215f-4ac0-9505-0bc2980bc3eb/volumes"
Nov 03 10:53:08.399943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:08.396753    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"]
Nov 03 10:53:08.399943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:08.397056    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy" containerID="cri-o://e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1" gracePeriod=30
Nov 03 10:53:13.211386 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:13.211177    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:53:14.141274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.141244    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:53:14.278720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.278636    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-credential-socket\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.278720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.278682    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-envoy\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.278728    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-data\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.278816    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-certs\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.278879    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-socket\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.278959    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istiod-ca-cert\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.278981    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-credential-socket" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.279058    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-token\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-token\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.279106    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-data" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.279145    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-podinfo\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.279235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.279146    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-certs" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:14.279677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.279248    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-kube-api-access-jbq6n\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-kube-api-access-jbq6n\") pod \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\" (UID: \"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356\") "
Nov 03 10:53:14.279677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.279232    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-socket" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:14.279677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.279399    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istiod-ca-cert" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:14.280126 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.280100    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.280216 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.280133    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.280216 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.280149    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.280216 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.280163    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.280216 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.280179    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.281461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.281433    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-token" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:14.282077 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.282051    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-envoy" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:14.282175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.282110    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-podinfo" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:53:14.285057 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.285031    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-kube-api-access-jbq6n" pod "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" (UID: "eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356"). InnerVolumeSpecName "kube-api-access-jbq6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:14.322223 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.322186    5393 generic.go:356] "Generic (PLEG): container finished" podID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerID="e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1" exitCode=0
Nov 03 10:53:14.322355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.322249    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"
Nov 03 10:53:14.322355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.322258    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" event={"ID":"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356","Type":"ContainerDied","Data":"e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1"}
Nov 03 10:53:14.322355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.322319    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr" event={"ID":"eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356","Type":"ContainerDied","Data":"34312f566b227bd1176717f8371c642fc8642cb22ec4498a89ca02e38f0f0117"}
Nov 03 10:53:14.322355 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.322341    5393 scope.go:122] "RemoveContainer" containerID="e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1"
Nov 03 10:53:14.333897 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.333874    5393 scope.go:122] "RemoveContainer" containerID="027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671"
Nov 03 10:53:14.343446 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.343430    5393 scope.go:122] "RemoveContainer" containerID="e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1"
Nov 03 10:53:14.343761 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:14.343740    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1\": container with ID starting with e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1 not found: ID does not exist" containerID="e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1"
Nov 03 10:53:14.343880 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.343769    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1"} err="failed to get container status \"e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1\": rpc error: code = NotFound desc = could not find container \"e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1\": container with ID starting with e08e5946c3d0da5431e0ae37cc3b9bce413dec4b86ffaceb8723f3c0dff7bcb1 not found: ID does not exist"
Nov 03 10:53:14.343880 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.343787    5393 scope.go:122] "RemoveContainer" containerID="027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671"
Nov 03 10:53:14.344060 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:14.344038    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671\": container with ID starting with 027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671 not found: ID does not exist" containerID="027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671"
Nov 03 10:53:14.344120 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.344066    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671"} err="failed to get container status \"027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671\": rpc error: code = NotFound desc = could not find container \"027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671\": container with ID starting with 027f48d59b18755ae47ed039daccb6b851c997e4bb1c53601e5203fb7723c671 not found: ID does not exist"
Nov 03 10:53:14.346532 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.346508    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"]
Nov 03 10:53:14.349362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.349344    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a51d55-openshift-default-685585d7df-8jzxr"]
Nov 03 10:53:14.382335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.382307    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.382335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.382331    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.382532 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.382345    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:14.382532 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:14.382355    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbq6n\" (UniqueName: \"kubernetes.io/projected/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356-kube-api-access-jbq6n\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:16.231099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:16.231067    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" path="/var/lib/kubelet/pods/eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356/volumes"
Nov 03 10:53:18.390673 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:18.387933    5393 generic.go:356] "Generic (PLEG): container finished" podID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerID="ecc7f2dff91d965b837ae2c8402c05a355bbcd64eeb9722e8922f8eec2876de0" exitCode=0
Nov 03 10:53:18.390673 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:18.388056    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" event={"ID":"5f8cc142-76a0-4c09-85bb-e5230e83bdbc","Type":"ContainerDied","Data":"ecc7f2dff91d965b837ae2c8402c05a355bbcd64eeb9722e8922f8eec2876de0"}
Nov 03 10:53:18.390673 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:18.388585    5393 scope.go:122] "RemoveContainer" containerID="ecc7f2dff91d965b837ae2c8402c05a355bbcd64eeb9722e8922f8eec2876de0"
Nov 03 10:53:18.392098 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:18.392054    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:18.392271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:18.392153    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:18.392271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:18.392195    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:19.413164 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:19.413132    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" event={"ID":"5f8cc142-76a0-4c09-85bb-e5230e83bdbc","Type":"ContainerStarted","Data":"1f24b4dd83faac2aafcbd40eacfcb654ff9b23ef8a8216329a4c21c31f3ab741"}
Nov 03 10:53:19.477877 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:19.477840    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:53:19.478111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:19.478080    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 10:53:19.478182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:19.478135    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 10:53:20.484482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:20.484445    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 10:53:20.485156 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:20.485121    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 10:53:21.144172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:21.144114    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"]
Nov 03 10:53:21.479899 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:21.479813    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 10:53:21.480182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:21.480149    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 10:53:22.478228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:22.478193    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 10:53:22.478620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:22.478243    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 10:53:23.477980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:23.477912    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 10:53:23.478259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:23.478002    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.464790    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"]
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465659    5393 cpu_manager.go:414] "containerMap: removing container" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465678    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465700    5393 cpu_manager.go:414] "containerMap: removing container" podUID="829f3652-eb44-4271-acf2-076be8e109bf" containerName="payload-pre-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465708    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829f3652-eb44-4271-acf2-076be8e109bf" containerName="payload-pre-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465734    5393 cpu_manager.go:414] "containerMap: removing container" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465742    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465764    5393 cpu_manager.go:414] "containerMap: removing container" podUID="58bad0a0-d5f1-469d-b11a-26af1f55fe16" containerName="payload-pre-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465771    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bad0a0-d5f1-469d-b11a-26af1f55fe16" containerName="payload-pre-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465794    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2fadd0de-215f-4ac0-9505-0bc2980bc3eb" containerName="revoke-keys"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465801    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fadd0de-215f-4ac0-9505-0bc2980bc3eb" containerName="revoke-keys"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465840    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5afc2511-2821-4359-a513-63a2f2aaa5b6" containerName="payload-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465848    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afc2511-2821-4359-a513-63a2f2aaa5b6" containerName="payload-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465856    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c01721d7-ef7a-40d9-af85-62b1bfa78f97" containerName="payload-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.465863    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01721d7-ef7a-40d9-af85-62b1bfa78f97" containerName="payload-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.466027    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bad0a0-d5f1-469d-b11a-26af1f55fe16" containerName="payload-pre-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.466042    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fadd0de-215f-4ac0-9505-0bc2980bc3eb" containerName="revoke-keys"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.466057    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea688a05-d023-4385-8f9c-54ceac47a320" containerName="istio-proxy"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.466072    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afc2511-2821-4359-a513-63a2f2aaa5b6" containerName="payload-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.466082    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01721d7-ef7a-40d9-af85-62b1bfa78f97" containerName="payload-processing"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.466095    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy"
Nov 03 10:53:24.466423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.466102    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="829f3652-eb44-4271-acf2-076be8e109bf" containerName="payload-pre-processing"
Nov 03 10:53:24.472764 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.470431    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.479163 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.477426    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 03 10:53:24.479163 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.477478    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 03 10:53:24.501088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.501058    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-18264e-openshift-default-dockercfg-b7b9p\""
Nov 03 10:53:24.520003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.519971    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"]
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.592940    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-credential-socket\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.593010    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-podinfo\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.593052    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-envoy\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.593088    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-certs\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.593123    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48beb4a5-a3c0-46b3-916c-f7685067032b-istiod-ca-cert\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.593156    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-token\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.593220    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-data\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.593248    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bvs\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-kube-api-access-44bvs\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.608533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.593418    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-socket\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.625521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.625488    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:53:24.720989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.702801    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-credential-socket\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.720989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.702856    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-podinfo\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.720989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.702898    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-envoy\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.720989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.702947    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-certs\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.720989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.702973    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48beb4a5-a3c0-46b3-916c-f7685067032b-istiod-ca-cert\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.720989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.703001    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-token\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.720989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.703050    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-data\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.720989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.703075    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bvs\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-kube-api-access-44bvs\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.722951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.722048    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-certs\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.722951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.722742    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48beb4a5-a3c0-46b3-916c-f7685067032b-istiod-ca-cert\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.742943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.738957    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-data\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.742943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.739244    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-socket\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.742943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.740399    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-socket\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.747942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.745657    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-envoy\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.747942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.746266    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-credential-socket\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.792942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.785475    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-token\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.801948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.795859    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-podinfo\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.801948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.798540    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bvs\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-kube-api-access-44bvs\") pod \"e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") " pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:24.837384 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:24.837350    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:25.041157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.041065    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"]
Nov 03 10:53:25.064053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.064024    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a9a12f-kkd6n"]
Nov 03 10:53:25.361581 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:53:25.361549    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48beb4a5_a3c0_46b3_916c_f7685067032b.slice/crio-d031756a5dfcde9b6e395f42454ca07f80cab5f2911e74a1965c8116800273d4 WatchSource:0}: Error finding container d031756a5dfcde9b6e395f42454ca07f80cab5f2911e74a1965c8116800273d4: Status 404 returned error can't find the container with id d031756a5dfcde9b6e395f42454ca07f80cab5f2911e74a1965c8116800273d4
Nov 03 10:53:25.362284 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.362252    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"]
Nov 03 10:53:25.364388 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.364351    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:25.364473 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.364431    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:25.364515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.364474    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:25.485225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.484292    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" event={"ID":"48beb4a5-a3c0-46b3-916c-f7685067032b","Type":"ContainerStarted","Data":"d031756a5dfcde9b6e395f42454ca07f80cab5f2911e74a1965c8116800273d4"}
Nov 03 10:53:25.516639 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.514837    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:53:25.521802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:25.521673    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:53:26.145096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.145047    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6" podUID="5a5eaa9d-97f2-4aea-aa45-264fecef58c1" containerName="payload-pre-processing" containerID="cri-o://0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05" gracePeriod=25
Nov 03 10:53:26.256638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.256602    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23750ab7-5579-47ce-9168-3e02effb1e3d" path="/var/lib/kubelet/pods/23750ab7-5579-47ce-9168-3e02effb1e3d/volumes"
Nov 03 10:53:26.420672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.420650    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:53:26.480193 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.480159    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-kube-api-access-wpnjn\" (UniqueName: \"kubernetes.io/projected/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-kube-api-access-wpnjn\") pod \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\" (UID: \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\") "
Nov 03 10:53:26.480374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.480361    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-plugins-config-volume\") pod \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\" (UID: \"5a5eaa9d-97f2-4aea-aa45-264fecef58c1\") "
Nov 03 10:53:26.480730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.480686    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-plugins-config-volume" pod "5a5eaa9d-97f2-4aea-aa45-264fecef58c1" (UID: "5a5eaa9d-97f2-4aea-aa45-264fecef58c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:26.480877 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.480862    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:26.482560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.482538    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-kube-api-access-wpnjn" pod "5a5eaa9d-97f2-4aea-aa45-264fecef58c1" (UID: "5a5eaa9d-97f2-4aea-aa45-264fecef58c1"). InnerVolumeSpecName "kube-api-access-wpnjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:26.492388 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.492358    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" event={"ID":"48beb4a5-a3c0-46b3-916c-f7685067032b","Type":"ContainerStarted","Data":"1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550"}
Nov 03 10:53:26.494097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.494076    5393 generic.go:356] "Generic (PLEG): container finished" podID="5a5eaa9d-97f2-4aea-aa45-264fecef58c1" containerID="0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05" exitCode=0
Nov 03 10:53:26.494187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.494119    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"
Nov 03 10:53:26.494187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.494147    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6" event={"ID":"5a5eaa9d-97f2-4aea-aa45-264fecef58c1","Type":"ContainerDied","Data":"0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05"}
Nov 03 10:53:26.494187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.494184    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6" event={"ID":"5a5eaa9d-97f2-4aea-aa45-264fecef58c1","Type":"ContainerDied","Data":"ef97468918dfa4a318f3699b03b48aab86856311e3474a0334cc6f01d8d0ef6f"}
Nov 03 10:53:26.494289 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.494204    5393 scope.go:122] "RemoveContainer" containerID="0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05"
Nov 03 10:53:26.504929 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.504896    5393 scope.go:122] "RemoveContainer" containerID="0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05"
Nov 03 10:53:26.505221 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:26.505204    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05\": container with ID starting with 0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05 not found: ID does not exist" containerID="0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05"
Nov 03 10:53:26.505289 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.505236    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05"} err="failed to get container status \"0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05\": rpc error: code = NotFound desc = could not find container \"0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05\": container with ID starting with 0acf511a41f2630a8f17aa4949a9d598ade7ce9e87607a3342536cae3be11b05 not found: ID does not exist"
Nov 03 10:53:26.517107 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.515442    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" podStartSLOduration=2.515425231 podStartE2EDuration="2.515425231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:26.51006236 +0000 UTC m=+1450.414916802" watchObservedRunningTime="2026-11-03 10:53:26.515425231 +0000 UTC m=+1450.420279673"
Nov 03 10:53:26.528318 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.528291    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"]
Nov 03 10:53:26.532734 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.532706    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6"]
Nov 03 10:53:26.582739 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.582709    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpnjn\" (UniqueName: \"kubernetes.io/projected/5a5eaa9d-97f2-4aea-aa45-264fecef58c1-kube-api-access-wpnjn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:26.838421 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.838325    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:26.844507 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:26.844478    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:27.408302 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:27.408255    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-08c392-6674df55cf-88tj6" podUID="5a5eaa9d-97f2-4aea-aa45-264fecef58c1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.127:50052\" within 1s: context deadline exceeded"
Nov 03 10:53:27.408521 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:53:27.408329    5393 logging.go:55] [core] [Channel #271 SubChannel #272]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.127:50052", ServerName: "10.134.0.127:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.127:50052: operation was canceled"
Nov 03 10:53:27.501816 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:27.501787    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:27.502729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:27.502712    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:53:28.231663 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:28.231627    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5eaa9d-97f2-4aea-aa45-264fecef58c1" path="/var/lib/kubelet/pods/5a5eaa9d-97f2-4aea-aa45-264fecef58c1/volumes"
Nov 03 10:53:30.028119 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.028080    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"]
Nov 03 10:53:30.028851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.028828    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5a5eaa9d-97f2-4aea-aa45-264fecef58c1" containerName="payload-pre-processing"
Nov 03 10:53:30.028970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.028852    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5eaa9d-97f2-4aea-aa45-264fecef58c1" containerName="payload-pre-processing"
Nov 03 10:53:30.029068 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.029053    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5eaa9d-97f2-4aea-aa45-264fecef58c1" containerName="payload-pre-processing"
Nov 03 10:53:30.029132 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.029077    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeaca4fd-b4cd-4e53-8854-5cd4c1bd8356" containerName="istio-proxy"
Nov 03 10:53:30.032018 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.031992    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:30.040538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.040512    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-18264e-dockercfg-46hd4\""
Nov 03 10:53:30.040673 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.040591    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-18264e\""
Nov 03 10:53:30.042881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.042862    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"]
Nov 03 10:53:30.061283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.061247    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"]
Nov 03 10:53:30.064887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.064855    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:30.073442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.073399    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"]
Nov 03 10:53:30.158116 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.158074    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f352c916-a60e-4f29-a4b8-0b45a1944089-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv\" (UID: \"f352c916-a60e-4f29-a4b8-0b45a1944089\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:30.158308 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.158172    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b438e276-513f-4f60-8824-a7eb88bc9116-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9\" (UID: \"b438e276-513f-4f60-8824-a7eb88bc9116\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:30.158498 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.158474    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8gzf\" (UniqueName: \"kubernetes.io/projected/f352c916-a60e-4f29-a4b8-0b45a1944089-kube-api-access-m8gzf\") pod \"payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv\" (UID: \"f352c916-a60e-4f29-a4b8-0b45a1944089\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:30.158572 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.158535    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdr89\" (UniqueName: \"kubernetes.io/projected/b438e276-513f-4f60-8824-a7eb88bc9116-kube-api-access-bdr89\") pod \"payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9\" (UID: \"b438e276-513f-4f60-8824-a7eb88bc9116\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:30.270012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.269899    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8gzf\" (UniqueName: \"kubernetes.io/projected/f352c916-a60e-4f29-a4b8-0b45a1944089-kube-api-access-m8gzf\") pod \"payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv\" (UID: \"f352c916-a60e-4f29-a4b8-0b45a1944089\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:30.270012 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.269994    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdr89\" (UniqueName: \"kubernetes.io/projected/b438e276-513f-4f60-8824-a7eb88bc9116-kube-api-access-bdr89\") pod \"payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9\" (UID: \"b438e276-513f-4f60-8824-a7eb88bc9116\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:30.270250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.270029    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f352c916-a60e-4f29-a4b8-0b45a1944089-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv\" (UID: \"f352c916-a60e-4f29-a4b8-0b45a1944089\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:30.270250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.270069    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b438e276-513f-4f60-8824-a7eb88bc9116-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9\" (UID: \"b438e276-513f-4f60-8824-a7eb88bc9116\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:30.271181 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.271149    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b438e276-513f-4f60-8824-a7eb88bc9116-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9\" (UID: \"b438e276-513f-4f60-8824-a7eb88bc9116\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:30.271806 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.271781    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f352c916-a60e-4f29-a4b8-0b45a1944089-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv\" (UID: \"f352c916-a60e-4f29-a4b8-0b45a1944089\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:30.324394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.300409    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdr89\" (UniqueName: \"kubernetes.io/projected/b438e276-513f-4f60-8824-a7eb88bc9116-kube-api-access-bdr89\") pod \"payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9\" (UID: \"b438e276-513f-4f60-8824-a7eb88bc9116\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:30.339944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.336845    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8gzf\" (UniqueName: \"kubernetes.io/projected/f352c916-a60e-4f29-a4b8-0b45a1944089-kube-api-access-m8gzf\") pod \"payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv\" (UID: \"f352c916-a60e-4f29-a4b8-0b45a1944089\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:30.353997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.353208    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:30.390323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.390293    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:30.810702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.810619    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"]
Nov 03 10:53:30.820636 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:53:30.820582    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf352c916_a60e_4f29_a4b8_0b45a1944089.slice/crio-9780b9e6600e14d952bd2dc8f7f7e54180ddb335538247c49e87465636f82e3b WatchSource:0}: Error finding container 9780b9e6600e14d952bd2dc8f7f7e54180ddb335538247c49e87465636f82e3b: Status 404 returned error can't find the container with id 9780b9e6600e14d952bd2dc8f7f7e54180ddb335538247c49e87465636f82e3b
Nov 03 10:53:30.825412 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:30.822280    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"]
Nov 03 10:53:30.826513 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:53:30.826122    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb438e276_513f_4f60_8824_a7eb88bc9116.slice/crio-80b16566a8748afd381f31368b5b05901ac567a7add604699a24d170950a4ab8 WatchSource:0}: Error finding container 80b16566a8748afd381f31368b5b05901ac567a7add604699a24d170950a4ab8: Status 404 returned error can't find the container with id 80b16566a8748afd381f31368b5b05901ac567a7add604699a24d170950a4ab8
Nov 03 10:53:31.674765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:31.674268    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9" event={"ID":"b438e276-513f-4f60-8824-a7eb88bc9116","Type":"ContainerStarted","Data":"8aadf6d18b521ee4685d12039388953e7ae154bcde6a6194216cb9f98df588ce"}
Nov 03 10:53:31.674765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:31.674381    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:31.674765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:31.674429    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9" event={"ID":"b438e276-513f-4f60-8824-a7eb88bc9116","Type":"ContainerStarted","Data":"80b16566a8748afd381f31368b5b05901ac567a7add604699a24d170950a4ab8"}
Nov 03 10:53:31.676084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:31.676060    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv" event={"ID":"f352c916-a60e-4f29-a4b8-0b45a1944089","Type":"ContainerStarted","Data":"267aba9dfa64981ebda42919887a1a908ae10a93ccb4ea1ca5feb0de146fd059"}
Nov 03 10:53:31.676223 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:31.676104    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv" event={"ID":"f352c916-a60e-4f29-a4b8-0b45a1944089","Type":"ContainerStarted","Data":"9780b9e6600e14d952bd2dc8f7f7e54180ddb335538247c49e87465636f82e3b"}
Nov 03 10:53:31.676223 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:31.676149    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:31.692210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:31.692140    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9" podStartSLOduration=1.692122859 podStartE2EDuration="1.692122859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:31.690439642 +0000 UTC m=+1455.595294084" watchObservedRunningTime="2026-11-03 10:53:31.692122859 +0000 UTC m=+1455.596977301"
Nov 03 10:53:31.707929 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:31.707862    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv" podStartSLOduration=1.707848206 podStartE2EDuration="1.707848206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:31.705279161 +0000 UTC m=+1455.610133599" watchObservedRunningTime="2026-11-03 10:53:31.707848206 +0000 UTC m=+1455.612702646"
Nov 03 10:53:36.425092 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:36.425055    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"]
Nov 03 10:53:36.426027 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:36.425965    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx" podUID="6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" containerName="maas-api" containerID="cri-o://0521e06f79f4fe6a1f01d67d05c3359a4cfd0d107449a80dbea0c56db6423dd4" gracePeriod=30
Nov 03 10:53:36.847399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:36.847366    5393 generic.go:356] "Generic (PLEG): container finished" podID="6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" containerID="0521e06f79f4fe6a1f01d67d05c3359a4cfd0d107449a80dbea0c56db6423dd4" exitCode=0
Nov 03 10:53:36.847889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:36.847731    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx" event={"ID":"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b","Type":"ContainerDied","Data":"0521e06f79f4fe6a1f01d67d05c3359a4cfd0d107449a80dbea0c56db6423dd4"}
Nov 03 10:53:37.045845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.045815    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:53:37.120651 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.120549    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-metrics-certs\") pod \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") "
Nov 03 10:53:37.120826 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.120713    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls\") pod \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") "
Nov 03 10:53:37.120826 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.120800    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-kube-api-access-ltlnp\" (UniqueName: \"kubernetes.io/projected/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-kube-api-access-ltlnp\") pod \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") "
Nov 03 10:53:37.120996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.120888    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-openshift-service-ca\") pod \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\" (UID: \"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b\") "
Nov 03 10:53:37.121580 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.121531    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-openshift-service-ca" pod "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" (UID: "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:37.121899 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.121874    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:37.129127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.129076    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-metrics-certs" pod "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" (UID: "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:37.129127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.129076    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-kube-api-access-ltlnp" pod "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" (UID: "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b"). InnerVolumeSpecName "kube-api-access-ltlnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:37.133066 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.133037    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls" pod "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" (UID: "6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:37.222799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.222762    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:37.222799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.222796    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:37.222799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.222807    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltlnp\" (UniqueName: \"kubernetes.io/projected/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b-kube-api-access-ltlnp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:37.857442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.857408    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"
Nov 03 10:53:37.857442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.857409    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx" event={"ID":"6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b","Type":"ContainerDied","Data":"dd600e80606c6ef1984cbda8765e8279d179f17e1fdefe154029310109d6b486"}
Nov 03 10:53:37.858050 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.857543    5393 scope.go:122] "RemoveContainer" containerID="0521e06f79f4fe6a1f01d67d05c3359a4cfd0d107449a80dbea0c56db6423dd4"
Nov 03 10:53:37.881412 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.881390    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"]
Nov 03 10:53:37.885799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:37.885779    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa420c-5f85d4d6fd-vxjgx"]
Nov 03 10:53:38.231786 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:38.231711    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" path="/var/lib/kubelet/pods/6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b/volumes"
Nov 03 10:53:42.721944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:42.715146    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:42.728948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:42.724434    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:43.427373 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:43.427326    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:53:44.425484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:44.425452    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"]
Nov 03 10:53:44.426321 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:44.426273    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy" containerID="cri-o://496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183" gracePeriod=30
Nov 03 10:53:46.527398 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.527359    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"]
Nov 03 10:53:46.529117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.529087    5393 cpu_manager.go:414] "containerMap: removing container" podUID="6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" containerName="maas-api"
Nov 03 10:53:46.529317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.529301    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" containerName="maas-api"
Nov 03 10:53:46.529578 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.529563    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e21a02c-c9d8-4ab2-b689-f2b16bf4d03b" containerName="maas-api"
Nov 03 10:53:46.553940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.545563    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.553940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.546609    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"]
Nov 03 10:53:46.557414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.557302    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-35afa6-openshift-default-dockercfg-dkjch\""
Nov 03 10:53:46.619546 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619496    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-credential-socket\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.619756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619556    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-podinfo\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.619756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619615    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnc4\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-kube-api-access-ncnc4\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.619756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619660    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-socket\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.619756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619735    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-token\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.619989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619763    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-data\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.619989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619792    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istiod-ca-cert\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.619989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619881    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-certs\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.619989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.619910    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-envoy\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.728910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.728859    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-certs\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.729189 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.729165    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-envoy\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.729423 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.729407    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-credential-socket\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.729554 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.729540    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-podinfo\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.729696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.729679    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnc4\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-kube-api-access-ncnc4\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.729844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.729829    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-socket\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.730054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.730034    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-token\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.730166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.730152    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-data\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.730408 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.730388    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istiod-ca-cert\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.731685 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.731664    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istiod-ca-cert\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.758946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.732424    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-certs\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.761595 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.761561    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-credential-socket\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.763609 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.763585    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-socket\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.764017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.763998    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-data\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.796787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.796611    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-envoy\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.797623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.797593    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnc4\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-kube-api-access-ncnc4\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.802484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.802426    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-token\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.821945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.818743    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-podinfo\") pod \"e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") " pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:46.907946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:46.878939    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:47.318282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:47.318246    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"]
Nov 03 10:53:47.320594 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:53:47.320547    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264e0847_a3ce_4bb2_ad7f_4a69a5b03bb7.slice/crio-f3fcc224069a23344462808d77d29a827bae6d04cae0a11a10c386de57008026 WatchSource:0}: Error finding container f3fcc224069a23344462808d77d29a827bae6d04cae0a11a10c386de57008026: Status 404 returned error can't find the container with id f3fcc224069a23344462808d77d29a827bae6d04cae0a11a10c386de57008026
Nov 03 10:53:47.323597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:47.323556    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:47.323792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:47.323770    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:47.323938 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:47.323904    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:48.290942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:48.284102    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" event={"ID":"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7","Type":"ContainerStarted","Data":"0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43"}
Nov 03 10:53:48.290942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:48.284231    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" event={"ID":"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7","Type":"ContainerStarted","Data":"f3fcc224069a23344462808d77d29a827bae6d04cae0a11a10c386de57008026"}
Nov 03 10:53:48.319936 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:48.319870    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podStartSLOduration=2.319854263 podStartE2EDuration="2.319854263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:48.318107057 +0000 UTC m=+1472.222961513" watchObservedRunningTime="2026-11-03 10:53:48.319854263 +0000 UTC m=+1472.224708703"
Nov 03 10:53:48.888841 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:48.888811    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:48.894912 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:48.894879    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:49.094715 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:49.094675    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"]
Nov 03 10:53:49.105230 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:49.105198    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"]
Nov 03 10:53:49.290912 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:49.290832    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:49.292052 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:49.292028    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:53:49.919943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:49.916617    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:53:50.116208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116173    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istiod-ca-cert\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.116426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116331    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-certs\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.116426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116383    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-token\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-token\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.116426 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116412    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-credential-socket\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.116619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116463    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-kube-api-access-n9jm5\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-kube-api-access-n9jm5\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.116619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116509    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-envoy\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.116619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116558    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-podinfo\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.116788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116633    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-socket\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.116788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116640    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istiod-ca-cert" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:50.116788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116667    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-certs" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:50.116788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116677    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-data\") pod \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\" (UID: \"f78765ce-ebc8-4d04-8ae6-721dddd8ba72\") "
Nov 03 10:53:50.117045 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.116860    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-data" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:50.117106 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.117073    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-credential-socket" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:50.117480 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.117451    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-socket" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:50.117587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.117511    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.117587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.117530    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.117587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.117543    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.117587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.117556    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.119768 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.119744    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-envoy" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:53:50.119884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.119793    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-kube-api-access-n9jm5" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "kube-api-access-n9jm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:50.120005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.119975    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-token" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:50.120094 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.120074    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-podinfo" pod "f78765ce-ebc8-4d04-8ae6-721dddd8ba72" (UID: "f78765ce-ebc8-4d04-8ae6-721dddd8ba72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:53:50.218802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.218759    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9jm5\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-kube-api-access-n9jm5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.218802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.218802    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.219029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.218817    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.219029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.218830    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.219029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.218847    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f78765ce-ebc8-4d04-8ae6-721dddd8ba72-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:50.300150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.300114    5393 generic.go:356] "Generic (PLEG): container finished" podID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerID="496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183" exitCode=0
Nov 03 10:53:50.300636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.300182    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"
Nov 03 10:53:50.300636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.300192    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" event={"ID":"f78765ce-ebc8-4d04-8ae6-721dddd8ba72","Type":"ContainerDied","Data":"496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183"}
Nov 03 10:53:50.300636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.300253    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft" event={"ID":"f78765ce-ebc8-4d04-8ae6-721dddd8ba72","Type":"ContainerDied","Data":"8f21b40529913ed14a9e5e59d8860ad5c70ceac3df1d9bc4f3eee515bede8a6a"}
Nov 03 10:53:50.300636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.300280    5393 scope.go:122] "RemoveContainer" containerID="496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183"
Nov 03 10:53:50.312149 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.312129    5393 scope.go:122] "RemoveContainer" containerID="8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1"
Nov 03 10:53:50.322820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.322763    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"]
Nov 03 10:53:50.325662 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.325637    5393 scope.go:122] "RemoveContainer" containerID="496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183"
Nov 03 10:53:50.326043 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:50.326023    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183\": container with ID starting with 496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183 not found: ID does not exist" containerID="496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183"
Nov 03 10:53:50.326127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.326050    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183"} err="failed to get container status \"496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183\": rpc error: code = NotFound desc = could not find container \"496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183\": container with ID starting with 496237cf4ed9c32317d87c14a0c64c4935c36715853f15d88ba1d271d900f183 not found: ID does not exist"
Nov 03 10:53:50.326127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.326068    5393 scope.go:122] "RemoveContainer" containerID="8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1"
Nov 03 10:53:50.326385 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:53:50.326370    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1\": container with ID starting with 8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1 not found: ID does not exist" containerID="8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1"
Nov 03 10:53:50.326436 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.326388    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1"} err="failed to get container status \"8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1\": rpc error: code = NotFound desc = could not find container \"8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1\": container with ID starting with 8e659cc270e720262d09f27bb700fb5788430f874ba172e2f8708080244458e1 not found: ID does not exist"
Nov 03 10:53:50.326932 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:50.326896    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa420c-openshift-default-575cf55575-5zsft"]
Nov 03 10:53:52.102570 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.102527    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"]
Nov 03 10:53:52.103157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.103139    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy"
Nov 03 10:53:52.103157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.103156    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy"
Nov 03 10:53:52.103290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.103171    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy"
Nov 03 10:53:52.103290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.103176    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy"
Nov 03 10:53:52.103392 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.103302    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy"
Nov 03 10:53:52.103392 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.103311    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" containerName="istio-proxy"
Nov 03 10:53:52.113127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.113098    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:52.116995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.116956    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"]
Nov 03 10:53:52.231591 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.231561    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6\" (UID: \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:52.385227 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.367388    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2h4\" (UniqueName: \"kubernetes.io/projected/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-kube-api-access-kj2h4\") pod \"payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6\" (UID: \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:52.385227 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.380214    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-35afa6\""
Nov 03 10:53:52.401634 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.385869    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78765ce-ebc8-4d04-8ae6-721dddd8ba72" path="/var/lib/kubelet/pods/f78765ce-ebc8-4d04-8ae6-721dddd8ba72/volumes"
Nov 03 10:53:52.401634 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.400318    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-35afa6-dockercfg-9jblj\""
Nov 03 10:53:52.471121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.471076    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6\" (UID: \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:52.471330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.471170    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2h4\" (UniqueName: \"kubernetes.io/projected/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-kube-api-access-kj2h4\") pod \"payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6\" (UID: \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:52.471868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.471846    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6\" (UID: \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:52.479381 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.479357    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2h4\" (UniqueName: \"kubernetes.io/projected/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-kube-api-access-kj2h4\") pod \"payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6\" (UID: \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:52.734007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:52.733254    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:53.027072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.027045    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"]
Nov 03 10:53:53.029045 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:53:53.029013    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98372fc1_b5f0_4444_a6f5_7d3a216cd88b.slice/crio-0b99beabcae029b896b2c631cdecaef487d29e2638d5230b98cb7813f7b56c4f WatchSource:0}: Error finding container 0b99beabcae029b896b2c631cdecaef487d29e2638d5230b98cb7813f7b56c4f: Status 404 returned error can't find the container with id 0b99beabcae029b896b2c631cdecaef487d29e2638d5230b98cb7813f7b56c4f
Nov 03 10:53:53.412933 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.412894    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6" event={"ID":"98372fc1-b5f0-4444-a6f5-7d3a216cd88b","Type":"ContainerStarted","Data":"5e06e5ffee6ca23f9464d6df507240d6ef74861560121864798247757a7ce328"}
Nov 03 10:53:53.413473 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.412991    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:53:53.413473 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.413013    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6" event={"ID":"98372fc1-b5f0-4444-a6f5-7d3a216cd88b","Type":"ContainerStarted","Data":"0b99beabcae029b896b2c631cdecaef487d29e2638d5230b98cb7813f7b56c4f"}
Nov 03 10:53:53.414964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.414910    5393 generic.go:356] "Generic (PLEG): container finished" podID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerID="48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f" exitCode=0
Nov 03 10:53:53.414964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.414946    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" event={"ID":"5d4f9c29-b268-4c8f-be71-f398b2978ba4","Type":"ContainerDied","Data":"48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f"}
Nov 03 10:53:53.415354 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.415339    5393 scope.go:122] "RemoveContainer" containerID="48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f"
Nov 03 10:53:53.416833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.416802    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:53.416930 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.416871    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:53.416930 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.416903    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:53:53.430038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.429988    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6" podStartSLOduration=1.429973466 podStartE2EDuration="1.429973466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:53.42805844 +0000 UTC m=+1477.332912882" watchObservedRunningTime="2026-11-03 10:53:53.429973466 +0000 UTC m=+1477.334827907"
Nov 03 10:53:53.987680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.987645    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"]
Nov 03 10:53:53.991734 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:53.991717    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:54.001129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.001106    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"]
Nov 03 10:53:54.001535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.001495    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-18264e-dockercfg-bmlpj\""
Nov 03 10:53:54.119942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.115317    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv" podUID="f352c916-a60e-4f29-a4b8-0b45a1944089" containerName="payload-pre-processing" containerID="cri-o://267aba9dfa64981ebda42919887a1a908ae10a93ccb4ea1ca5feb0de146fd059" gracePeriod=25
Nov 03 10:53:54.152945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.144808    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba214d7f-08af-498e-a14f-f3151a3423d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2\" (UID: \"ba214d7f-08af-498e-a14f-f3151a3423d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:54.152945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.144893    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq257\" (UniqueName: \"kubernetes.io/projected/ba214d7f-08af-498e-a14f-f3151a3423d3-kube-api-access-dq257\") pod \"payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2\" (UID: \"ba214d7f-08af-498e-a14f-f3151a3423d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:54.152945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.151410    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9" podUID="b438e276-513f-4f60-8824-a7eb88bc9116" containerName="payload-processing" containerID="cri-o://8aadf6d18b521ee4685d12039388953e7ae154bcde6a6194216cb9f98df588ce" gracePeriod=25
Nov 03 10:53:54.272942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.252346    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba214d7f-08af-498e-a14f-f3151a3423d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2\" (UID: \"ba214d7f-08af-498e-a14f-f3151a3423d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:54.272942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.252426    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq257\" (UniqueName: \"kubernetes.io/projected/ba214d7f-08af-498e-a14f-f3151a3423d3-kube-api-access-dq257\") pod \"payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2\" (UID: \"ba214d7f-08af-498e-a14f-f3151a3423d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:54.272942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.253666    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba214d7f-08af-498e-a14f-f3151a3423d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2\" (UID: \"ba214d7f-08af-498e-a14f-f3151a3423d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:54.282115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.282031    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq257\" (UniqueName: \"kubernetes.io/projected/ba214d7f-08af-498e-a14f-f3151a3423d3-kube-api-access-dq257\") pod \"payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2\" (UID: \"ba214d7f-08af-498e-a14f-f3151a3423d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:54.308442 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.308179    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:54.435454 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.435419    5393 generic.go:356] "Generic (PLEG): container finished" podID="f352c916-a60e-4f29-a4b8-0b45a1944089" containerID="267aba9dfa64981ebda42919887a1a908ae10a93ccb4ea1ca5feb0de146fd059" exitCode=0
Nov 03 10:53:54.435843 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.435627    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv" event={"ID":"f352c916-a60e-4f29-a4b8-0b45a1944089","Type":"ContainerDied","Data":"267aba9dfa64981ebda42919887a1a908ae10a93ccb4ea1ca5feb0de146fd059"}
Nov 03 10:53:54.439117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.438331    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" event={"ID":"5d4f9c29-b268-4c8f-be71-f398b2978ba4","Type":"ContainerStarted","Data":"bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2"}
Nov 03 10:53:54.440438 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.440414    5393 generic.go:356] "Generic (PLEG): container finished" podID="b438e276-513f-4f60-8824-a7eb88bc9116" containerID="8aadf6d18b521ee4685d12039388953e7ae154bcde6a6194216cb9f98df588ce" exitCode=0
Nov 03 10:53:54.440517 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.440447    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9" event={"ID":"b438e276-513f-4f60-8824-a7eb88bc9116","Type":"ContainerDied","Data":"8aadf6d18b521ee4685d12039388953e7ae154bcde6a6194216cb9f98df588ce"}
Nov 03 10:53:54.473789 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.473749    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:53:54.474573 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.474544    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:53:54.474703 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.474608    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 10:53:54.644196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.644161    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:54.646585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.646565    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:54.676270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.675383    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"]
Nov 03 10:53:54.678252 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:53:54.678224    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba214d7f_08af_498e_a14f_f3151a3423d3.slice/crio-a3f3f0455000622cfb570178ad10aba172aa6488f02b7569fdf14f89941221f2 WatchSource:0}: Error finding container a3f3f0455000622cfb570178ad10aba172aa6488f02b7569fdf14f89941221f2: Status 404 returned error can't find the container with id a3f3f0455000622cfb570178ad10aba172aa6488f02b7569fdf14f89941221f2
Nov 03 10:53:54.705937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.701247    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"]
Nov 03 10:53:54.705937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.703499    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b438e276-513f-4f60-8824-a7eb88bc9116" containerName="payload-processing"
Nov 03 10:53:54.705937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.703528    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b438e276-513f-4f60-8824-a7eb88bc9116" containerName="payload-processing"
Nov 03 10:53:54.705937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.703838    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f352c916-a60e-4f29-a4b8-0b45a1944089" containerName="payload-pre-processing"
Nov 03 10:53:54.705937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.703848    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f352c916-a60e-4f29-a4b8-0b45a1944089" containerName="payload-pre-processing"
Nov 03 10:53:54.705937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.704724    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f352c916-a60e-4f29-a4b8-0b45a1944089" containerName="payload-pre-processing"
Nov 03 10:53:54.705937 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.704755    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b438e276-513f-4f60-8824-a7eb88bc9116" containerName="payload-processing"
Nov 03 10:53:54.725051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.725012    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"]
Nov 03 10:53:54.725051 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.725045    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:54.729053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.728990    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:53:54.729565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.729540    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:53:54.784259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.778618    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f352c916-a60e-4f29-a4b8-0b45a1944089-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f352c916-a60e-4f29-a4b8-0b45a1944089-plugins-config-volume\") pod \"f352c916-a60e-4f29-a4b8-0b45a1944089\" (UID: \"f352c916-a60e-4f29-a4b8-0b45a1944089\") "
Nov 03 10:53:54.784259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.778667    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b438e276-513f-4f60-8824-a7eb88bc9116-kube-api-access-bdr89\" (UniqueName: \"kubernetes.io/projected/b438e276-513f-4f60-8824-a7eb88bc9116-kube-api-access-bdr89\") pod \"b438e276-513f-4f60-8824-a7eb88bc9116\" (UID: \"b438e276-513f-4f60-8824-a7eb88bc9116\") "
Nov 03 10:53:54.784259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.778727    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f352c916-a60e-4f29-a4b8-0b45a1944089-kube-api-access-m8gzf\" (UniqueName: \"kubernetes.io/projected/f352c916-a60e-4f29-a4b8-0b45a1944089-kube-api-access-m8gzf\") pod \"f352c916-a60e-4f29-a4b8-0b45a1944089\" (UID: \"f352c916-a60e-4f29-a4b8-0b45a1944089\") "
Nov 03 10:53:54.784259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.778833    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b438e276-513f-4f60-8824-a7eb88bc9116-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b438e276-513f-4f60-8824-a7eb88bc9116-plugins-config-volume\") pod \"b438e276-513f-4f60-8824-a7eb88bc9116\" (UID: \"b438e276-513f-4f60-8824-a7eb88bc9116\") "
Nov 03 10:53:54.784755 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.784696    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b438e276-513f-4f60-8824-a7eb88bc9116-plugins-config-volume" pod "b438e276-513f-4f60-8824-a7eb88bc9116" (UID: "b438e276-513f-4f60-8824-a7eb88bc9116"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:54.785785 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.785743    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f352c916-a60e-4f29-a4b8-0b45a1944089-plugins-config-volume" pod "f352c916-a60e-4f29-a4b8-0b45a1944089" (UID: "f352c916-a60e-4f29-a4b8-0b45a1944089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:54.791674 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.791648    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f352c916-a60e-4f29-a4b8-0b45a1944089-kube-api-access-m8gzf" pod "f352c916-a60e-4f29-a4b8-0b45a1944089" (UID: "f352c916-a60e-4f29-a4b8-0b45a1944089"). InnerVolumeSpecName "kube-api-access-m8gzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:54.841904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.838611    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b438e276-513f-4f60-8824-a7eb88bc9116-kube-api-access-bdr89" pod "b438e276-513f-4f60-8824-a7eb88bc9116" (UID: "b438e276-513f-4f60-8824-a7eb88bc9116"). InnerVolumeSpecName "kube-api-access-bdr89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:54.899939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.892964    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:54.899939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.893054    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjfs\" (UniqueName: \"kubernetes.io/projected/e00349a3-007d-45f0-a826-85ee0e5a1742-kube-api-access-mcjfs\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:54.899939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.893119    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:54.899939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.893258    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f352c916-a60e-4f29-a4b8-0b45a1944089-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:54.899939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.893274    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdr89\" (UniqueName: \"kubernetes.io/projected/b438e276-513f-4f60-8824-a7eb88bc9116-kube-api-access-bdr89\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:54.899939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.893288    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8gzf\" (UniqueName: \"kubernetes.io/projected/f352c916-a60e-4f29-a4b8-0b45a1944089-kube-api-access-m8gzf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:54.899939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.893301    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b438e276-513f-4f60-8824-a7eb88bc9116-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:54.994269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.994227    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:54.994613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.994586    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjfs\" (UniqueName: \"kubernetes.io/projected/e00349a3-007d-45f0-a826-85ee0e5a1742-kube-api-access-mcjfs\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:54.994806 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.994788    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:54.995431 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:54.995388    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:55.002739 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.002708    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:55.009691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.009659    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjfs\" (UniqueName: \"kubernetes.io/projected/e00349a3-007d-45f0-a826-85ee0e5a1742-kube-api-access-mcjfs\") pod \"maas-api-revoke-sub-8d179392-v5lgg\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:55.106387 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.104520    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:55.413889 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.413858    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"]
Nov 03 10:53:55.416424 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:53:55.416395    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00349a3_007d_45f0_a826_85ee0e5a1742.slice/crio-ebb4dc55f25d8cf64ad01b466e590ae585f203b359493236ee27bcb91cc076db WatchSource:0}: Error finding container ebb4dc55f25d8cf64ad01b466e590ae585f203b359493236ee27bcb91cc076db: Status 404 returned error can't find the container with id ebb4dc55f25d8cf64ad01b466e590ae585f203b359493236ee27bcb91cc076db
Nov 03 10:53:55.449636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.449604    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2" event={"ID":"ba214d7f-08af-498e-a14f-f3151a3423d3","Type":"ContainerStarted","Data":"0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520"}
Nov 03 10:53:55.450067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.449672    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2" event={"ID":"ba214d7f-08af-498e-a14f-f3151a3423d3","Type":"ContainerStarted","Data":"a3f3f0455000622cfb570178ad10aba172aa6488f02b7569fdf14f89941221f2"}
Nov 03 10:53:55.450619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.450597    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:53:55.451958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.451909    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9" event={"ID":"b438e276-513f-4f60-8824-a7eb88bc9116","Type":"ContainerDied","Data":"80b16566a8748afd381f31368b5b05901ac567a7add604699a24d170950a4ab8"}
Nov 03 10:53:55.452050 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.451981    5393 scope.go:122] "RemoveContainer" containerID="8aadf6d18b521ee4685d12039388953e7ae154bcde6a6194216cb9f98df588ce"
Nov 03 10:53:55.452050 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.451945    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"
Nov 03 10:53:55.453220 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.453192    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg" event={"ID":"e00349a3-007d-45f0-a826-85ee0e5a1742","Type":"ContainerStarted","Data":"ebb4dc55f25d8cf64ad01b466e590ae585f203b359493236ee27bcb91cc076db"}
Nov 03 10:53:55.454680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.454651    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"
Nov 03 10:53:55.454680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.454668    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv" event={"ID":"f352c916-a60e-4f29-a4b8-0b45a1944089","Type":"ContainerDied","Data":"9780b9e6600e14d952bd2dc8f7f7e54180ddb335538247c49e87465636f82e3b"}
Nov 03 10:53:55.462290 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.462273    5393 scope.go:122] "RemoveContainer" containerID="267aba9dfa64981ebda42919887a1a908ae10a93ccb4ea1ca5feb0de146fd059"
Nov 03 10:53:55.468793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.468747    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2" podStartSLOduration=2.468731992 podStartE2EDuration="2.468731992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:55.466549914 +0000 UTC m=+1479.371404357" watchObservedRunningTime="2026-11-03 10:53:55.468731992 +0000 UTC m=+1479.373586435"
Nov 03 10:53:55.473451 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.473427    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:53:55.473533 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.473492    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 10:53:55.505054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.504978    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"]
Nov 03 10:53:55.510338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.510304    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-18264e-7d45bb8b6f-cnlm9"]
Nov 03 10:53:55.520443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.520411    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"]
Nov 03 10:53:55.526002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:55.525975    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-65b8fb9867-bgwxv"]
Nov 03 10:53:56.277947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:56.273894    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b438e276-513f-4f60-8824-a7eb88bc9116" path="/var/lib/kubelet/pods/b438e276-513f-4f60-8824-a7eb88bc9116/volumes"
Nov 03 10:53:56.302957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:56.295836    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f352c916-a60e-4f29-a4b8-0b45a1944089" path="/var/lib/kubelet/pods/f352c916-a60e-4f29-a4b8-0b45a1944089/volumes"
Nov 03 10:53:56.464260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:56.464222    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg" event={"ID":"e00349a3-007d-45f0-a826-85ee0e5a1742","Type":"ContainerStarted","Data":"e587d7057d1060290d0ffd3e5f46de8d009ddbff4fc0a55808e5a7a7b02a7203"}
Nov 03 10:53:56.474255 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:56.474225    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:53:56.474422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:56.474274    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 10:53:56.493953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:56.481101    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg" podStartSLOduration=2.481083995 podStartE2EDuration="2.481083995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:53:56.479992502 +0000 UTC m=+1480.384846944" watchObservedRunningTime="2026-11-03 10:53:56.481083995 +0000 UTC m=+1480.385938436"
Nov 03 10:53:57.473495 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:57.473465    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:53:57.474015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:57.473524    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 10:53:57.475501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:57.475352    5393 generic.go:356] "Generic (PLEG): container finished" podID="e00349a3-007d-45f0-a826-85ee0e5a1742" containerID="e587d7057d1060290d0ffd3e5f46de8d009ddbff4fc0a55808e5a7a7b02a7203" exitCode=0
Nov 03 10:53:57.476965 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:57.475393    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg" event={"ID":"e00349a3-007d-45f0-a826-85ee0e5a1742","Type":"ContainerDied","Data":"e587d7057d1060290d0ffd3e5f46de8d009ddbff4fc0a55808e5a7a7b02a7203"}
Nov 03 10:53:58.473293 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.473255    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:53:58.473482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.473307    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 10:53:58.810962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.810907    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:58.846470 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.846430    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-openshift-ca\") pod \"e00349a3-007d-45f0-a826-85ee0e5a1742\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") "
Nov 03 10:53:58.847131 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.847105    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00349a3-007d-45f0-a826-85ee0e5a1742-kube-api-access-mcjfs\" (UniqueName: \"kubernetes.io/projected/e00349a3-007d-45f0-a826-85ee0e5a1742-kube-api-access-mcjfs\") pod \"e00349a3-007d-45f0-a826-85ee0e5a1742\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") "
Nov 03 10:53:58.847343 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.847104    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-openshift-ca" pod "e00349a3-007d-45f0-a826-85ee0e5a1742" (UID: "e00349a3-007d-45f0-a826-85ee0e5a1742"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:53:58.847483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.847324    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-xks-ca\") pod \"e00349a3-007d-45f0-a826-85ee0e5a1742\" (UID: \"e00349a3-007d-45f0-a826-85ee0e5a1742\") "
Nov 03 10:53:58.848244 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.848213    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:58.849999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.849896    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00349a3-007d-45f0-a826-85ee0e5a1742-kube-api-access-mcjfs" pod "e00349a3-007d-45f0-a826-85ee0e5a1742" (UID: "e00349a3-007d-45f0-a826-85ee0e5a1742"). InnerVolumeSpecName "kube-api-access-mcjfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:53:58.853954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.850447    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-xks-ca" pod "e00349a3-007d-45f0-a826-85ee0e5a1742" (UID: "e00349a3-007d-45f0-a826-85ee0e5a1742"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:53:58.860618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.860599    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:53:58.950101 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.950064    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcjfs\" (UniqueName: \"kubernetes.io/projected/e00349a3-007d-45f0-a826-85ee0e5a1742-kube-api-access-mcjfs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:58.950101 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:58.950100    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e00349a3-007d-45f0-a826-85ee0e5a1742-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:53:59.473605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:59.473570    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:53:59.473784 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:59.473623    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 10:53:59.514174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:59.514148    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"
Nov 03 10:53:59.514348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:59.514181    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg" event={"ID":"e00349a3-007d-45f0-a826-85ee0e5a1742","Type":"ContainerDied","Data":"ebb4dc55f25d8cf64ad01b466e590ae585f203b359493236ee27bcb91cc076db"}
Nov 03 10:53:59.514348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:53:59.514229    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb4dc55f25d8cf64ad01b466e590ae585f203b359493236ee27bcb91cc076db"
Nov 03 10:54:00.473854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:00.473279    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 10:54:00.473854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:00.473336    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 10:54:01.822944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:01.822090    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:01.959946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:01.958967    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:54:01.984945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:01.975264    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:54:02.334945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:02.331029    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:02.643666 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:02.638025    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:02.643666 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:02.638437    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:54:02.676454 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:02.675037    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:02.796943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:02.796113    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:02.825949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:02.822740    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:54:02.825949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:02.823281    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:54:02.969455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:02.969348    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:03.216568 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.050236    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:03.216568 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.131449    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:54:03.356003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.346817    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:03.356003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.348186    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:54:03.648131 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.648029    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:03.678977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.648243    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:03.678977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.650046    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:54:03.678977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.650128    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:54:03.834944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.830069    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:03.834944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.830459    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 10:54:03.834944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.831120    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:54:03.873396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.873368    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:54:03.909042 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:03.905661    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:54:04.041545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.041508    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:04.267298 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.266463    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:04.267298 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.266953    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:54:04.347304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.347248    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:04.358686 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.358567    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:54:04.359499 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.359479    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:54:04.533261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.533161    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:54:04.664991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.660951    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:04.843515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.842055    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:04.869948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.865289    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:54:04.883420 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.883392    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:54:04.975616 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.975592    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:54:04.976999 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.976852    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:54:04.978515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:04.978498    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:54:07.478592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.478564    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:54:07.935291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.935254    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"]
Nov 03 10:54:07.935933 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.935893    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e00349a3-007d-45f0-a826-85ee0e5a1742" containerName="revoke-keys"
Nov 03 10:54:07.936301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.935933    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00349a3-007d-45f0-a826-85ee0e5a1742" containerName="revoke-keys"
Nov 03 10:54:07.936301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.936145    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00349a3-007d-45f0-a826-85ee0e5a1742" containerName="revoke-keys"
Nov 03 10:54:07.939631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.939611    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:07.943351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.943325    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:54:07.943459 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.943387    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:54:07.946841 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.946803    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"]
Nov 03 10:54:07.965840 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.965811    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:07.966056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.965968    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:07.966135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:07.966074    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8852\" (UniqueName: \"kubernetes.io/projected/61935e03-223a-410e-97bd-0bdebdb1b7e4-kube-api-access-h8852\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:08.067592 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:08.067560    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8852\" (UniqueName: \"kubernetes.io/projected/61935e03-223a-410e-97bd-0bdebdb1b7e4-kube-api-access-h8852\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:08.067757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:08.067738    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:08.067887 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:08.067869    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:08.068658 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:08.068631    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:08.070351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:08.070329    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:08.075629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:08.075611    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8852\" (UniqueName: \"kubernetes.io/projected/61935e03-223a-410e-97bd-0bdebdb1b7e4-kube-api-access-h8852\") pod \"maas-api-revoke-sub-6036a896-7cj5g\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:08.266082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:08.265999    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:08.545957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:08.545912    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"]
Nov 03 10:54:08.546508 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:54:08.546485    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61935e03_223a_410e_97bd_0bdebdb1b7e4.slice/crio-622cd8dc842a476b8bdc3af759542a8e8cc3ad9768ce58cc88d46cef1e492536 WatchSource:0}: Error finding container 622cd8dc842a476b8bdc3af759542a8e8cc3ad9768ce58cc88d46cef1e492536: Status 404 returned error can't find the container with id 622cd8dc842a476b8bdc3af759542a8e8cc3ad9768ce58cc88d46cef1e492536
Nov 03 10:54:09.045646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:09.045600    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g" event={"ID":"61935e03-223a-410e-97bd-0bdebdb1b7e4","Type":"ContainerStarted","Data":"0f1d6ba6bc3b7b46ee8b0331879fbefaa257ea27055bb41c82bf31215bdfdc85"}
Nov 03 10:54:09.045811 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:09.045716    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g" event={"ID":"61935e03-223a-410e-97bd-0bdebdb1b7e4","Type":"ContainerStarted","Data":"622cd8dc842a476b8bdc3af759542a8e8cc3ad9768ce58cc88d46cef1e492536"}
Nov 03 10:54:09.064986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:09.064856    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g" podStartSLOduration=2.06484178 podStartE2EDuration="2.06484178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:09.061225792 +0000 UTC m=+1492.966080232" watchObservedRunningTime="2026-11-03 10:54:09.06484178 +0000 UTC m=+1492.969696220"
Nov 03 10:54:10.056067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:10.056031    5393 generic.go:356] "Generic (PLEG): container finished" podID="61935e03-223a-410e-97bd-0bdebdb1b7e4" containerID="0f1d6ba6bc3b7b46ee8b0331879fbefaa257ea27055bb41c82bf31215bdfdc85" exitCode=0
Nov 03 10:54:10.056503 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:10.056095    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g" event={"ID":"61935e03-223a-410e-97bd-0bdebdb1b7e4","Type":"ContainerDied","Data":"0f1d6ba6bc3b7b46ee8b0331879fbefaa257ea27055bb41c82bf31215bdfdc85"}
Nov 03 10:54:11.199118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.199067    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"]
Nov 03 10:54:11.208519 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.208414    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.208721 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.208703    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"]
Nov 03 10:54:11.220249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.219482    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8bbb6e-openshift-default-dockercfg-8drsf\""
Nov 03 10:54:11.287976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.279605    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:11.299404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.298841    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-xks-ca\") pod \"61935e03-223a-410e-97bd-0bdebdb1b7e4\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") "
Nov 03 10:54:11.299404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299005    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-openshift-ca\") pod \"61935e03-223a-410e-97bd-0bdebdb1b7e4\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") "
Nov 03 10:54:11.299404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299276    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61935e03-223a-410e-97bd-0bdebdb1b7e4-kube-api-access-h8852\" (UniqueName: \"kubernetes.io/projected/61935e03-223a-410e-97bd-0bdebdb1b7e4-kube-api-access-h8852\") pod \"61935e03-223a-410e-97bd-0bdebdb1b7e4\" (UID: \"61935e03-223a-410e-97bd-0bdebdb1b7e4\") "
Nov 03 10:54:11.299713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299503    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-socket\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.299713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299547    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-certs\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.299713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299574    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-token\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.299713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299627    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-credential-socket\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.299953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299799    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-data\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.299953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299941    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5mq\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-kube-api-access-cv5mq\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.300072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.299973    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-podinfo\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.300215 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.300167    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-envoy\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.300405 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.300273    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istiod-ca-cert\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.301441 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.301417    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-openshift-ca" pod "61935e03-223a-410e-97bd-0bdebdb1b7e4" (UID: "61935e03-223a-410e-97bd-0bdebdb1b7e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:11.307646 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.307617    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-xks-ca" pod "61935e03-223a-410e-97bd-0bdebdb1b7e4" (UID: "61935e03-223a-410e-97bd-0bdebdb1b7e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:54:11.309757 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.309729    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61935e03-223a-410e-97bd-0bdebdb1b7e4-kube-api-access-h8852" pod "61935e03-223a-410e-97bd-0bdebdb1b7e4" (UID: "61935e03-223a-410e-97bd-0bdebdb1b7e4"). InnerVolumeSpecName "kube-api-access-h8852". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410153    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-data\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410270    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5mq\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-kube-api-access-cv5mq\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410302    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-podinfo\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410433    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-envoy\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410501    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istiod-ca-cert\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410548    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-socket\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410586    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-certs\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410611    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-token\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410652    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-credential-socket\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410812    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8852\" (UniqueName: \"kubernetes.io/projected/61935e03-223a-410e-97bd-0bdebdb1b7e4-kube-api-access-h8852\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410834    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.410852    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61935e03-223a-410e-97bd-0bdebdb1b7e4-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.411314    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-credential-socket\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.442605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.411601    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-data\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.565710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.458351    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-socket\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.565710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.459005    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-certs\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.565710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.459609    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istiod-ca-cert\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.570553 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.570448    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-envoy\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.573063 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.573010    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-token\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.596583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.596505    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5mq\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-kube-api-access-cv5mq\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.609803 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.609720    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-podinfo\") pod \"e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") " pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:11.841018 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:11.840906    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:12.077948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.077809    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g" event={"ID":"61935e03-223a-410e-97bd-0bdebdb1b7e4","Type":"ContainerDied","Data":"622cd8dc842a476b8bdc3af759542a8e8cc3ad9768ce58cc88d46cef1e492536"}
Nov 03 10:54:12.077948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.077874    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="622cd8dc842a476b8bdc3af759542a8e8cc3ad9768ce58cc88d46cef1e492536"
Nov 03 10:54:12.085949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.082394    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"
Nov 03 10:54:12.141449 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.140722    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"]
Nov 03 10:54:12.141449 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:54:12.140836    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821a9850_2564_4179_8aeb_d80fb0e4aaf0.slice/crio-e527cfec27a0389ef4fe9bdc63b0d33a889ee0d177f405bc9e075b7dc7c8b240 WatchSource:0}: Error finding container e527cfec27a0389ef4fe9bdc63b0d33a889ee0d177f405bc9e075b7dc7c8b240: Status 404 returned error can't find the container with id e527cfec27a0389ef4fe9bdc63b0d33a889ee0d177f405bc9e075b7dc7c8b240
Nov 03 10:54:12.143905 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.143868    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:12.144008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.143977    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:12.144055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.144022    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:12.457360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.455877    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"]
Nov 03 10:54:12.457360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.456861    5393 cpu_manager.go:414] "containerMap: removing container" podUID="61935e03-223a-410e-97bd-0bdebdb1b7e4" containerName="revoke-keys"
Nov 03 10:54:12.457360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.456882    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61935e03-223a-410e-97bd-0bdebdb1b7e4" containerName="revoke-keys"
Nov 03 10:54:12.457360 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.457156    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="61935e03-223a-410e-97bd-0bdebdb1b7e4" containerName="revoke-keys"
Nov 03 10:54:12.463849 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.463812    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.474970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.464765    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"]
Nov 03 10:54:12.527175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.527142    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-343e30-openshift-default-dockercfg-h5ph5\""
Nov 03 10:54:12.627893 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.627837    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-credential-socket\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.628266 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.628237    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-certs\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.628509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.628491    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-socket\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.634080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.634054    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-envoy\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.634312 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.634296    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-data\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.634501 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.634487    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-podinfo\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.634764 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.634743    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istiod-ca-cert\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.635023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.635006    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-token\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.638942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.635192    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksd5s\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-kube-api-access-ksd5s\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.740983    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istiod-ca-cert\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.741087    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-token\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.741143    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksd5s\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-kube-api-access-ksd5s\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.741263    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-credential-socket\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.741286    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-certs\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.741346    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-socket\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.741385    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-envoy\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.741415    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-data\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.741449    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-podinfo\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.742370    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-credential-socket\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.743128    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istiod-ca-cert\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.743821    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-socket\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.760399    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-envoy\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.760804    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-data\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.767175    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-certs\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.772591    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksd5s\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-kube-api-access-ksd5s\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.787549    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-podinfo\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.798541 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.796268    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-token\") pod \"e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") " pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:12.800989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:12.800960    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:13.091444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.090246    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" event={"ID":"821a9850-2564-4179-8aeb-d80fb0e4aaf0","Type":"ContainerStarted","Data":"f2cda572e13813b7d379c79c5e0eb4503c1e6f76a38d9d2a300580565bd32753"}
Nov 03 10:54:13.091444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.090318    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" event={"ID":"821a9850-2564-4179-8aeb-d80fb0e4aaf0","Type":"ContainerStarted","Data":"e527cfec27a0389ef4fe9bdc63b0d33a889ee0d177f405bc9e075b7dc7c8b240"}
Nov 03 10:54:13.119788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.116029    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podStartSLOduration=2.116009502 podStartE2EDuration="2.116009502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:13.111691536 +0000 UTC m=+1497.016545979" watchObservedRunningTime="2026-11-03 10:54:13.116009502 +0000 UTC m=+1497.020863944"
Nov 03 10:54:13.192328 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:54:13.192288    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:54:13.258944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.257781    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": read tcp 10.134.0.2:34078->10.134.0.132:15021: read: connection reset by peer" start-of-body=
Nov 03 10:54:13.258944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.257842    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": read tcp 10.134.0.2:34078->10.134.0.132:15021: read: connection reset by peer"
Nov 03 10:54:13.389910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.373960    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"]
Nov 03 10:54:13.398970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.397490    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:13.398970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.397584    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:13.398970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.397629    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:13.841242 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.841205    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:13.847335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:13.847310    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:14.176689 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.176661    5393 generic.go:356] "Generic (PLEG): container finished" podID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerID="1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550" exitCode=0
Nov 03 10:54:14.176882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.176797    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" event={"ID":"48beb4a5-a3c0-46b3-916c-f7685067032b","Type":"ContainerDied","Data":"1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550"}
Nov 03 10:54:14.177269 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.177239    5393 scope.go:122] "RemoveContainer" containerID="1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550"
Nov 03 10:54:14.179080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.179058    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" event={"ID":"068911eb-8f9c-4e54-a792-ee9a96ff8cf7","Type":"ContainerStarted","Data":"1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4"}
Nov 03 10:54:14.179259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.179242    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" event={"ID":"068911eb-8f9c-4e54-a792-ee9a96ff8cf7","Type":"ContainerStarted","Data":"f49f0fc2cdde18a44f99f8da961cd6f5535a3478c275ef511d861d10b763d1b3"}
Nov 03 10:54:14.179395 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.179381    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:14.179504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.179099    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:14.179666 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.179636    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:14.179775 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.179682    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:14.180589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.180573    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:54:14.245792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.243632    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podStartSLOduration=2.243612579 podStartE2EDuration="2.243612579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:14.241378039 +0000 UTC m=+1498.146232482" watchObservedRunningTime="2026-11-03 10:54:14.243612579 +0000 UTC m=+1498.148467021"
Nov 03 10:54:14.801782 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.801734    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:14.807808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:14.807779    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:15.217973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:15.214930    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" event={"ID":"48beb4a5-a3c0-46b3-916c-f7685067032b","Type":"ContainerStarted","Data":"6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42"}
Nov 03 10:54:15.221953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:15.220359    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:15.225299 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:15.223655    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:54:15.620810 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:15.620763    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"]
Nov 03 10:54:15.621053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:15.620984    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy" containerID="cri-o://073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b" gracePeriod=30
Nov 03 10:54:15.837942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:15.837899    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:54:15.838294 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:15.838066    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 10:54:15.838294 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:15.838102    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 10:54:16.832682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:16.832648    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"]
Nov 03 10:54:16.837814 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:16.837790    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:16.838128 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:16.838098    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 10:54:16.838267 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:16.838158    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 10:54:16.844348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:16.844329    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8bbb6e-dockercfg-trsc5\""
Nov 03 10:54:16.847107 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:16.847083    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"]
Nov 03 10:54:16.923597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:16.923560    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8bbb6e\""
Nov 03 10:54:17.235332 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.233040    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hhl\" (UniqueName: \"kubernetes.io/projected/a3066ccb-d8dd-40a1-834b-c2f855805204-kube-api-access-n5hhl\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx\" (UID: \"a3066ccb-d8dd-40a1-834b-c2f855805204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:17.235701 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.235674    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3066ccb-d8dd-40a1-834b-c2f855805204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx\" (UID: \"a3066ccb-d8dd-40a1-834b-c2f855805204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:17.337177 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.337149    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hhl\" (UniqueName: \"kubernetes.io/projected/a3066ccb-d8dd-40a1-834b-c2f855805204-kube-api-access-n5hhl\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx\" (UID: \"a3066ccb-d8dd-40a1-834b-c2f855805204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:17.337407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.337393    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3066ccb-d8dd-40a1-834b-c2f855805204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx\" (UID: \"a3066ccb-d8dd-40a1-834b-c2f855805204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:17.338838 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.338811    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3066ccb-d8dd-40a1-834b-c2f855805204-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx\" (UID: \"a3066ccb-d8dd-40a1-834b-c2f855805204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:17.342478 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.342456    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:54:17.346556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.346534    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:54:17.353659 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.353637    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hhl\" (UniqueName: \"kubernetes.io/projected/a3066ccb-d8dd-40a1-834b-c2f855805204-kube-api-access-n5hhl\") pod \"payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx\" (UID: \"a3066ccb-d8dd-40a1-834b-c2f855805204\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:17.459852 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.459822    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:17.775023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.774991    5393 scope.go:122] "RemoveContainer" containerID="00be0a73d5eac571ba84c4bd8c7cb765924f10235a02806f83a226e21b1c9487"
Nov 03 10:54:17.812931 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.812878    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"]
Nov 03 10:54:17.838296 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.838274    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 10:54:17.838788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:17.838328    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 10:54:18.336151 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:18.336108    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx" event={"ID":"a3066ccb-d8dd-40a1-834b-c2f855805204","Type":"ContainerStarted","Data":"614e3729185310bac704d469c218ad2439e65df53a05a0d261be0f44013cda14"}
Nov 03 10:54:18.336312 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:18.336178    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx" event={"ID":"a3066ccb-d8dd-40a1-834b-c2f855805204","Type":"ContainerStarted","Data":"93bc7837fef0ef3d0e22996221780e2681da9fdecf17e0159b540f40c94d40db"}
Nov 03 10:54:18.337283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:18.337261    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:18.354907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:18.354850    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx" podStartSLOduration=2.354833142 podStartE2EDuration="2.354833142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:18.352970939 +0000 UTC m=+1502.257825381" watchObservedRunningTime="2026-11-03 10:54:18.354833142 +0000 UTC m=+1502.259687586"
Nov 03 10:54:18.838578 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:18.838539    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 10:54:18.839107 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:18.838600    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 10:54:19.838417 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:19.838388    5393 patch_prober.go:29] interesting pod/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Nov 03 10:54:19.838598 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:19.838448    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Nov 03 10:54:20.845161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.845124    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:54:20.846557 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.846447    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:54:20.847289 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.847244    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:54:20.924205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.924132    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:54:20.977608 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.977549    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-kube-api-access-58lpm\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-kube-api-access-58lpm\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.977608 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.977605    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-credential-socket\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.978033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.977644    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-certs\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.978033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.977708    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-podinfo\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.978033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.977770    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-data\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.978033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.977911    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-socket\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.978033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.977993    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-token\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-token\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.978443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.978081    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-envoy\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.978443 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.978217    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b295b2f8-4f1e-48e8-af11-40899d42e907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b295b2f8-4f1e-48e8-af11-40899d42e907-istiod-ca-cert\") pod \"b295b2f8-4f1e-48e8-af11-40899d42e907\" (UID: \"b295b2f8-4f1e-48e8-af11-40899d42e907\") "
Nov 03 10:54:20.979475 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.979447    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-data" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:54:20.979637 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.979620    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-socket" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:54:20.980970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.980908    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b295b2f8-4f1e-48e8-af11-40899d42e907-istiod-ca-cert" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:20.983139 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.983025    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-credential-socket" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:54:20.984031 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.983994    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-certs" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:54:20.989946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.989911    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-podinfo" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:54:20.991070 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.991048    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-token" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:20.991198 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.991123    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-envoy" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:54:20.991359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:20.991342    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-kube-api-access-58lpm" pod "b295b2f8-4f1e-48e8-af11-40899d42e907" (UID: "b295b2f8-4f1e-48e8-af11-40899d42e907"). InnerVolumeSpecName "kube-api-access-58lpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:21.080690 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080650    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.080690 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080686    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.080940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080699    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.080940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080711    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b295b2f8-4f1e-48e8-af11-40899d42e907-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.080940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080725    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lpm\" (UniqueName: \"kubernetes.io/projected/b295b2f8-4f1e-48e8-af11-40899d42e907-kube-api-access-58lpm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.080940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080738    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.080940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080749    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.080940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080761    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.080940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.080772    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b295b2f8-4f1e-48e8-af11-40899d42e907-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:21.441811 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.441754    5393 generic.go:356] "Generic (PLEG): container finished" podID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerID="073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b" exitCode=0
Nov 03 10:54:21.442055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.441846    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"
Nov 03 10:54:21.442055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.441837    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" event={"ID":"b295b2f8-4f1e-48e8-af11-40899d42e907","Type":"ContainerDied","Data":"073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b"}
Nov 03 10:54:21.442055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.442019    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn" event={"ID":"b295b2f8-4f1e-48e8-af11-40899d42e907","Type":"ContainerDied","Data":"97c02643512d2ad65ad99039dc367de17402c15f8b0a5070557ac158f9e29c0c"}
Nov 03 10:54:21.442240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.442064    5393 scope.go:122] "RemoveContainer" containerID="073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b"
Nov 03 10:54:21.456031 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.455991    5393 scope.go:122] "RemoveContainer" containerID="073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b"
Nov 03 10:54:21.456336 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:54:21.456313    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b\": container with ID starting with 073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b not found: ID does not exist" containerID="073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b"
Nov 03 10:54:21.456404 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.456344    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b"} err="failed to get container status \"073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b\": rpc error: code = NotFound desc = could not find container \"073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b\": container with ID starting with 073b2d0a37928bab5f63e56aae75b0eeb93093d282aaec18e7e4ebe7ba368a9b not found: ID does not exist"
Nov 03 10:54:21.467122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.467070    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"]
Nov 03 10:54:21.470319 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:21.470299    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-5c5zn"]
Nov 03 10:54:22.231813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:22.231776    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" path="/var/lib/kubelet/pods/b295b2f8-4f1e-48e8-af11-40899d42e907/volumes"
Nov 03 10:54:25.848944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:25.843538    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 10:54:25.891827 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:25.891033    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 10:54:25.908059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:25.908020    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:54:25.963212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:25.962301    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:25.976069 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:25.975711    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:54:25.976069 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:25.975775    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:25.977877 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:25.977516    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:25.994961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:25.984990    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.045947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.045157    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.117341 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.117171    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.358015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.357022    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.438000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.435012    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:54:26.438000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.435089    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.647109 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.647053    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.845010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.844167    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.845010 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.844877    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:54:26.920969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.901072    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.920969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.904490    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:54:26.920969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.909959    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:26.920969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:26.910309    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:54:27.076414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.016080    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:27.076414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.016762    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:54:27.076414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.045458    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:27.076414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.064418    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:54:27.076414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.064480    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:54:27.105000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.104384    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:54:27.150953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.140028    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:54:27.162951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.159050    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:27.162951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.159461    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:54:27.549116 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.547325    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:54:27.597991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.556554    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:54:27.597991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.557837    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:27.597991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.558211    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:54:27.597991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.558687    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:54:27.652946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:27.634379    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:54:28.053196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.053168    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"]
Nov 03 10:54:28.053759 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.053734    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy"
Nov 03 10:54:28.053838 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.053761    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy"
Nov 03 10:54:28.053912 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.053901    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b295b2f8-4f1e-48e8-af11-40899d42e907" containerName="istio-proxy"
Nov 03 10:54:28.059262 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.059239    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:28.071617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.071590    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-343e30-dockercfg-cz5c7\""
Nov 03 10:54:28.072031 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.072012    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-343e30\""
Nov 03 10:54:28.072647 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.072622    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"]
Nov 03 10:54:28.124929 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.124884    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"]
Nov 03 10:54:28.129393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.129374    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:28.206819 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.206780    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"]
Nov 03 10:54:28.251830 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.251788    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8lzb\" (UniqueName: \"kubernetes.io/projected/3e789eb9-9cf4-42ed-81a7-dc4097993f11-kube-api-access-b8lzb\") pod \"payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7\" (UID: \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:28.258009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.251876    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc\" (UID: \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:28.258009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.251993    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ktbm\" (UniqueName: \"kubernetes.io/projected/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-kube-api-access-8ktbm\") pod \"payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc\" (UID: \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:28.258009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.252116    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e789eb9-9cf4-42ed-81a7-dc4097993f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7\" (UID: \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:28.355240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.355203    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8lzb\" (UniqueName: \"kubernetes.io/projected/3e789eb9-9cf4-42ed-81a7-dc4097993f11-kube-api-access-b8lzb\") pod \"payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7\" (UID: \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:28.355545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.355522    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc\" (UID: \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:28.355748 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.355729    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktbm\" (UniqueName: \"kubernetes.io/projected/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-kube-api-access-8ktbm\") pod \"payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc\" (UID: \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:28.356011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.355993    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e789eb9-9cf4-42ed-81a7-dc4097993f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7\" (UID: \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:28.357065 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.357039    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e789eb9-9cf4-42ed-81a7-dc4097993f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7\" (UID: \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:28.371801 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.358772    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc\" (UID: \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:28.470768 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.470733    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktbm\" (UniqueName: \"kubernetes.io/projected/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-kube-api-access-8ktbm\") pod \"payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc\" (UID: \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:28.494033 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.493992    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"]
Nov 03 10:54:28.579294 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.579231    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8lzb\" (UniqueName: \"kubernetes.io/projected/3e789eb9-9cf4-42ed-81a7-dc4097993f11-kube-api-access-b8lzb\") pod \"payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7\" (UID: \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:28.579460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.579417    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"]
Nov 03 10:54:28.579545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.579529    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.672351 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.672229    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.672769 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.672727    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.674085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.674060    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.674344 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.674326    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.674842 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.674667    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:28.684728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.684697    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.685151 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.685126    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbwfw\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-kube-api-access-qbwfw\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.685364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.685348    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.685526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.685511    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.685699 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.685686    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.761956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.747170    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:28.812956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.797875    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.815974 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.813634    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.816255    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.816306    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.816408    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbwfw\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-kube-api-access-qbwfw\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.816485    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.816560    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.816667    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.816775    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.817294    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-certs\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.815130    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-token\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.817735    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-credential-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.814443    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-socket\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.819309    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-data\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.819497    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-envoy\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.819969 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.819683    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istiod-ca-cert\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.822150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.822121    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-podinfo\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.830556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.830500    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbwfw\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-kube-api-access-qbwfw\") pod \"e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") " pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:28.919121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:28.919033    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:29.064115 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.064082    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"]
Nov 03 10:54:29.079342 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:54:29.079317    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e789eb9_9cf4_42ed_81a7_dc4097993f11.slice/crio-44ab52d53c64e5ccea492dd1e503ebd701fc62e7e8b7abe89a809a6a01a3637b WatchSource:0}: Error finding container 44ab52d53c64e5ccea492dd1e503ebd701fc62e7e8b7abe89a809a6a01a3637b: Status 404 returned error can't find the container with id 44ab52d53c64e5ccea492dd1e503ebd701fc62e7e8b7abe89a809a6a01a3637b
Nov 03 10:54:29.119122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.113722    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"]
Nov 03 10:54:29.220040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.220016    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"]
Nov 03 10:54:29.224964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.224712    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:29.224964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.224783    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:29.224964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.224835    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:29.720800 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.720760    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" event={"ID":"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7","Type":"ContainerStarted","Data":"b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8"}
Nov 03 10:54:29.721025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.720818    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" event={"ID":"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7","Type":"ContainerStarted","Data":"9f50cec797cfd8b27499e9019ac5684aeea4b76865520c59b2b835a32237e059"}
Nov 03 10:54:29.722731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.722700    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc" event={"ID":"d1937d6c-7aa5-4e49-bffa-18f54cb3aada","Type":"ContainerStarted","Data":"b12e465d984eb2bdbec9ace6bdef8a640bfbe8caaa000b58467144d446a27b72"}
Nov 03 10:54:29.722891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.722754    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc" event={"ID":"d1937d6c-7aa5-4e49-bffa-18f54cb3aada","Type":"ContainerStarted","Data":"8f2891694ad34e7c5f4b754261ab40547cd0f59c70fcd44510cceb82e4f628ea"}
Nov 03 10:54:29.722891 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.722799    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:29.736968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.736930    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7" event={"ID":"3e789eb9-9cf4-42ed-81a7-dc4097993f11","Type":"ContainerStarted","Data":"668f7ae8096c7f19595484c57bd2772923f48f12a4254939538b748f61ca2133"}
Nov 03 10:54:29.737108 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.737024    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7" event={"ID":"3e789eb9-9cf4-42ed-81a7-dc4097993f11","Type":"ContainerStarted","Data":"44ab52d53c64e5ccea492dd1e503ebd701fc62e7e8b7abe89a809a6a01a3637b"}
Nov 03 10:54:29.737108 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.737048    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:29.743519 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.743467    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" podStartSLOduration=1.7434504039999998 podStartE2EDuration="1.743450404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:29.740470611 +0000 UTC m=+1513.645325069" watchObservedRunningTime="2026-11-03 10:54:29.743450404 +0000 UTC m=+1513.648304846"
Nov 03 10:54:29.778943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.768039    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7" podStartSLOduration=1.7680186039999999 podStartE2EDuration="1.768018604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:29.758625965 +0000 UTC m=+1513.663480407" watchObservedRunningTime="2026-11-03 10:54:29.768018604 +0000 UTC m=+1513.672873046"
Nov 03 10:54:29.925979 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:29.920012    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:30.374643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:30.374617    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:30.392639 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:30.392578    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc" podStartSLOduration=2.392555967 podStartE2EDuration="2.392555967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:54:29.789234117 +0000 UTC m=+1513.694088559" watchObservedRunningTime="2026-11-03 10:54:30.392555967 +0000 UTC m=+1514.297410412"
Nov 03 10:54:30.922535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:30.922441    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 10:54:30.922535 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:30.922509    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 10:54:31.928691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:31.928654    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 10:54:31.929307 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:31.929276    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 10:54:32.919523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:32.919488    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 10:54:32.919705 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:32.919562    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 10:54:33.920092 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:33.920054    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 10:54:33.920494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:33.920146    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 10:54:34.919554 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:34.919516    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 10:54:34.919709 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:34.919571    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 10:54:35.333136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:35.333050    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:54:35.333136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:35.333116    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:35.919670 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:35.919630    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 10:54:35.919863 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:35.919714    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 10:54:37.427830 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.427797    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:37.430013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.429990    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:37.430825 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.430801    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 10:54:37.986356 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.986261    5393 generic.go:356] "Generic (PLEG): container finished" podID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerID="0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43" exitCode=0
Nov 03 10:54:37.986356 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.986341    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" event={"ID":"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7","Type":"ContainerDied","Data":"0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43"}
Nov 03 10:54:37.987171 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.987151    5393 scope.go:122] "RemoveContainer" containerID="0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43"
Nov 03 10:54:37.996545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.989722    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:37.996545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.989942    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:37.996545 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:37.990013    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:54:39.015365 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:39.015328    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" event={"ID":"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7","Type":"ContainerStarted","Data":"0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b"}
Nov 03 10:54:39.901945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:39.892308    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:54:39.901945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:39.892477    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 10:54:39.901945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:39.892524    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 10:54:40.671264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:40.671231    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"]
Nov 03 10:54:40.763853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:40.763821    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:54:40.765970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:40.765580    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:54:40.907044 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:40.906373    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 10:54:40.907044 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:40.906431    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 10:54:41.912956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:41.911449    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 10:54:41.912956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:41.911517    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 10:54:42.899945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:42.897514    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 10:54:42.899945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:42.897581    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 10:54:43.219609 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:54:43.219520    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:54:43.891340 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:43.891301    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body=
Nov 03 10:54:43.891657 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:43.891627    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused"
Nov 03 10:54:44.893435 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:44.893403    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:54:44.893870 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:44.893683    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:54:44.894584 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:44.894567    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:54:45.672259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:45.672215    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx" podUID="a3066ccb-d8dd-40a1-834b-c2f855805204" containerName="payload-pre-processing" containerID="cri-o://614e3729185310bac704d469c218ad2439e65df53a05a0d261be0f44013cda14" gracePeriod=25
Nov 03 10:54:46.165966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.165300    5393 generic.go:356] "Generic (PLEG): container finished" podID="a3066ccb-d8dd-40a1-834b-c2f855805204" containerID="614e3729185310bac704d469c218ad2439e65df53a05a0d261be0f44013cda14" exitCode=0
Nov 03 10:54:46.166586 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.166557    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx" event={"ID":"a3066ccb-d8dd-40a1-834b-c2f855805204","Type":"ContainerDied","Data":"614e3729185310bac704d469c218ad2439e65df53a05a0d261be0f44013cda14"}
Nov 03 10:54:46.251764 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.251727    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:46.310783 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.310379    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3066ccb-d8dd-40a1-834b-c2f855805204-kube-api-access-n5hhl\" (UniqueName: \"kubernetes.io/projected/a3066ccb-d8dd-40a1-834b-c2f855805204-kube-api-access-n5hhl\") pod \"a3066ccb-d8dd-40a1-834b-c2f855805204\" (UID: \"a3066ccb-d8dd-40a1-834b-c2f855805204\") "
Nov 03 10:54:46.311003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.310912    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3066ccb-d8dd-40a1-834b-c2f855805204-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3066ccb-d8dd-40a1-834b-c2f855805204-plugins-config-volume\") pod \"a3066ccb-d8dd-40a1-834b-c2f855805204\" (UID: \"a3066ccb-d8dd-40a1-834b-c2f855805204\") "
Nov 03 10:54:46.311540 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.311513    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3066ccb-d8dd-40a1-834b-c2f855805204-plugins-config-volume" pod "a3066ccb-d8dd-40a1-834b-c2f855805204" (UID: "a3066ccb-d8dd-40a1-834b-c2f855805204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:54:46.313119 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.313092    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3066ccb-d8dd-40a1-834b-c2f855805204-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:46.317218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.317150    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3066ccb-d8dd-40a1-834b-c2f855805204-kube-api-access-n5hhl" pod "a3066ccb-d8dd-40a1-834b-c2f855805204" (UID: "a3066ccb-d8dd-40a1-834b-c2f855805204"). InnerVolumeSpecName "kube-api-access-n5hhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:54:46.414847 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:46.414813    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5hhl\" (UniqueName: \"kubernetes.io/projected/a3066ccb-d8dd-40a1-834b-c2f855805204-kube-api-access-n5hhl\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:54:47.215515 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:47.215468    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx" event={"ID":"a3066ccb-d8dd-40a1-834b-c2f855805204","Type":"ContainerDied","Data":"93bc7837fef0ef3d0e22996221780e2681da9fdecf17e0159b540f40c94d40db"}
Nov 03 10:54:47.216035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:47.215585    5393 scope.go:122] "RemoveContainer" containerID="614e3729185310bac704d469c218ad2439e65df53a05a0d261be0f44013cda14"
Nov 03 10:54:47.216035 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:47.215772    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"
Nov 03 10:54:47.252867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:47.252839    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"]
Nov 03 10:54:47.257636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:47.257610    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8bbb6e-7b7f9c44d7-8mkzx"]
Nov 03 10:54:48.231049 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:48.231023    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3066ccb-d8dd-40a1-834b-c2f855805204" path="/var/lib/kubelet/pods/a3066ccb-d8dd-40a1-834b-c2f855805204/volumes"
Nov 03 10:54:49.564947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:49.561026    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:49.640161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:49.639019    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:49.648244 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:49.648204    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 10:54:50.006478 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.006432    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.052080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.052028    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.146190 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.145017    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 10:54:50.150113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.150072    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.560946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.549297    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.572023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.568392    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 10:54:50.572023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.569546    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:50.572023 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.569981    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:54:50.845249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.845133    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 03 10:54:50.848953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.846165    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:54:50.848953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:50.846602    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:54:51.015361 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.015315    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:51.023272 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.016130    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:54:51.113641 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.113588    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 10:54:51.131965 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.131933    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:54:51.155743 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.155701    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:51.164344 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.164309    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:54:51.166662 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.166629    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:51.481550 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.481471    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:54:51.578053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.578008    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 10:54:51.947520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.663770    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:51.947520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.691768    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 10:54:51.947520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.691990    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 10:54:51.947520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.727362    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:54:51.947520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.928297    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:54:51.947520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.928362    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:54:51.947520 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:51.928479    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:54:52.042411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.042371    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:52.165531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.156819    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:52.319003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.311509    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded"
Nov 03 10:54:52.495585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.495524    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:52.598366 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.598340    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 10:54:52.865117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.865015    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:52.899129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.899071    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:52.926944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.922572    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:52.926944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.922986    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 10:54:52.926944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.923644    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:54:52.944019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:52.928296    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:53.051760 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.051628    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 10:54:53.109207 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.109177    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 10:54:53.119977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.119868    5393 patch_prober.go:29] interesting pod/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:54:53.120239 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.120202    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:53.257605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.257564    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:53.506138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.501063    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:53.880482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.878232    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:53.880482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.878786    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:54:53.926168 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.926127    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:53.927556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.927523    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 10:54:53.932677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:53.932018    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:54.062523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:54.062003    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:54:54.170439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:54.170354    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 10:54:54.406946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:54.375342    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 10:54:54.406946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:54.375405    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 10:54:54.406946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:54.375440    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 10:54:54.456696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:54.456190    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 10:54:54.458307 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:54:54.458278    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:55:00.004645 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.004600    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"]
Nov 03 10:55:00.005229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.005213    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a3066ccb-d8dd-40a1-834b-c2f855805204" containerName="payload-pre-processing"
Nov 03 10:55:00.005229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.005228    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3066ccb-d8dd-40a1-834b-c2f855805204" containerName="payload-pre-processing"
Nov 03 10:55:00.005371 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.005362    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3066ccb-d8dd-40a1-834b-c2f855805204" containerName="payload-pre-processing"
Nov 03 10:55:00.041430 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.041391    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"]
Nov 03 10:55:00.041651 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.041632    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.046053 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.046029    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:55:00.047153 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.047131    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:55:00.190797 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.190757    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c84g\" (UniqueName: \"kubernetes.io/projected/ad9141e9-c8b0-4315-9197-235304be49c7-kube-api-access-7c84g\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.191030 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.190939    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.191182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.191163    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.225083 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.225051    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:55:00.225270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.225099    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:00.292627 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.292529    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c84g\" (UniqueName: \"kubernetes.io/projected/ad9141e9-c8b0-4315-9197-235304be49c7-kube-api-access-7c84g\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.292813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.292672    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.292952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.292904    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.293483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.293459    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.295633 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.295613    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.303116 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.303091    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c84g\" (UniqueName: \"kubernetes.io/projected/ad9141e9-c8b0-4315-9197-235304be49c7-kube-api-access-7c84g\") pod \"maas-api-revoke-sub-ed0b3e82-g2sbn\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.385340 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.385302    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:00.740867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:00.739965    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"]
Nov 03 10:55:01.259156 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:01.259066    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:55:01.259156 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:01.259132    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:01.590732 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:01.590649    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn" event={"ID":"ad9141e9-c8b0-4315-9197-235304be49c7","Type":"ContainerStarted","Data":"445f832df449b92a4c733c647b44a2b762ab2c33f962cfc56e797897923db830"}
Nov 03 10:55:01.597516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:01.590745    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn" event={"ID":"ad9141e9-c8b0-4315-9197-235304be49c7","Type":"ContainerStarted","Data":"f8618dfecd0efcc17f25670210c96a786cb531c717cb0df87e40d1b9c65f4dd2"}
Nov 03 10:55:01.609336 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:01.609291    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn" podStartSLOduration=2.609277623 podStartE2EDuration="2.609277623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:55:01.605388585 +0000 UTC m=+1545.510243026" watchObservedRunningTime="2026-11-03 10:55:01.609277623 +0000 UTC m=+1545.514132064"
Nov 03 10:55:02.599905 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.599871    5393 generic.go:356] "Generic (PLEG): container finished" podID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerID="1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4" exitCode=0
Nov 03 10:55:02.600490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.599953    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" event={"ID":"068911eb-8f9c-4e54-a792-ee9a96ff8cf7","Type":"ContainerDied","Data":"1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4"}
Nov 03 10:55:02.600564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.600496    5393 scope.go:122] "RemoveContainer" containerID="1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4"
Nov 03 10:55:02.601894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.601865    5393 generic.go:356] "Generic (PLEG): container finished" podID="ad9141e9-c8b0-4315-9197-235304be49c7" containerID="445f832df449b92a4c733c647b44a2b762ab2c33f962cfc56e797897923db830" exitCode=0
Nov 03 10:55:02.602008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.601963    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn" event={"ID":"ad9141e9-c8b0-4315-9197-235304be49c7","Type":"ContainerDied","Data":"445f832df449b92a4c733c647b44a2b762ab2c33f962cfc56e797897923db830"}
Nov 03 10:55:02.602364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.602338    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:55:02.602444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.602426    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:55:02.602494 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.602477    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:55:02.603930 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.603897    5393 generic.go:356] "Generic (PLEG): container finished" podID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerID="f2cda572e13813b7d379c79c5e0eb4503c1e6f76a38d9d2a300580565bd32753" exitCode=0
Nov 03 10:55:02.604018 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.603942    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" event={"ID":"821a9850-2564-4179-8aeb-d80fb0e4aaf0","Type":"ContainerDied","Data":"f2cda572e13813b7d379c79c5e0eb4503c1e6f76a38d9d2a300580565bd32753"}
Nov 03 10:55:02.604348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.604329    5393 scope.go:122] "RemoveContainer" containerID="f2cda572e13813b7d379c79c5e0eb4503c1e6f76a38d9d2a300580565bd32753"
Nov 03 10:55:02.605995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.605966    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:55:02.606100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.606045    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:55:02.606100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:02.606092    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:55:03.572717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:03.572679    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"]
Nov 03 10:55:03.658475 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:03.658262    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" event={"ID":"821a9850-2564-4179-8aeb-d80fb0e4aaf0","Type":"ContainerStarted","Data":"840440d2d455cfa3e949593c0df6a1231615bad4beabf4f4ef7b37f694944de9"}
Nov 03 10:55:03.663161 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:03.662327    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" event={"ID":"068911eb-8f9c-4e54-a792-ee9a96ff8cf7","Type":"ContainerStarted","Data":"2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e"}
Nov 03 10:55:03.805943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:03.803717    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:55:04.056941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.035677    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 10:55:04.056941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.035743    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 10:55:04.056941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.045734    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:55:04.102029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.092661    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 10:55:04.102029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.092731    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 10:55:04.447818 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.447789    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:04.544672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.544621    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9141e9-c8b0-4315-9197-235304be49c7-kube-api-access-7c84g\" (UniqueName: \"kubernetes.io/projected/ad9141e9-c8b0-4315-9197-235304be49c7-kube-api-access-7c84g\") pod \"ad9141e9-c8b0-4315-9197-235304be49c7\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") "
Nov 03 10:55:04.544886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.544731    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-xks-ca\") pod \"ad9141e9-c8b0-4315-9197-235304be49c7\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") "
Nov 03 10:55:04.544886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.544851    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-openshift-ca\") pod \"ad9141e9-c8b0-4315-9197-235304be49c7\" (UID: \"ad9141e9-c8b0-4315-9197-235304be49c7\") "
Nov 03 10:55:04.546007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.545978    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-openshift-ca" pod "ad9141e9-c8b0-4315-9197-235304be49c7" (UID: "ad9141e9-c8b0-4315-9197-235304be49c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:04.547814 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.547787    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9141e9-c8b0-4315-9197-235304be49c7-kube-api-access-7c84g" pod "ad9141e9-c8b0-4315-9197-235304be49c7" (UID: "ad9141e9-c8b0-4315-9197-235304be49c7"). InnerVolumeSpecName "kube-api-access-7c84g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:04.547951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.547867    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-xks-ca" pod "ad9141e9-c8b0-4315-9197-235304be49c7" (UID: "ad9141e9-c8b0-4315-9197-235304be49c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:04.647821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.647745    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c84g\" (UniqueName: \"kubernetes.io/projected/ad9141e9-c8b0-4315-9197-235304be49c7-kube-api-access-7c84g\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:04.647821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.647775    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:04.647821 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.647784    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9141e9-c8b0-4315-9197-235304be49c7-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:04.670631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.670599    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"
Nov 03 10:55:04.671136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.670623    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn" event={"ID":"ad9141e9-c8b0-4315-9197-235304be49c7","Type":"ContainerDied","Data":"f8618dfecd0efcc17f25670210c96a786cb531c717cb0df87e40d1b9c65f4dd2"}
Nov 03 10:55:04.671136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.670676    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8618dfecd0efcc17f25670210c96a786cb531c717cb0df87e40d1b9c65f4dd2"
Nov 03 10:55:04.805941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.804973    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 10:55:04.805941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.805036    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 10:55:04.865538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.865502    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 10:55:04.877410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:04.877365    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 10:55:05.805946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:05.803542    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 10:55:05.805946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:05.803615    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 10:55:05.841951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:05.841787    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 10:55:05.841951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:05.841855    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 10:55:06.837950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:06.836701    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 10:55:06.837950 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:06.836789    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 10:55:06.861753 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:06.861716    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 10:55:06.862083 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:06.862054    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 10:55:07.478014 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:07.477977    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"]
Nov 03 10:55:07.667782 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:07.667604    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"]
Nov 03 10:55:07.831081 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:07.811054    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 10:55:07.975897 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:07.975078    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 10:55:08.008274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:08.008231    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 10:55:08.008563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:08.008524    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 10:55:08.573009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:08.572968    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6" podUID="98372fc1-b5f0-4444-a6f5-7d3a216cd88b" containerName="payload-processing" containerID="cri-o://5e06e5ffee6ca23f9464d6df507240d6ef74861560121864798247757a7ce328" gracePeriod=25
Nov 03 10:55:08.815943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:08.815370    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 10:55:08.815943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:08.815434    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 10:55:08.842048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:08.841967    5393 patch_prober.go:29] interesting pod/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 10:55:08.842048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:08.842035    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 10:55:09.047966 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.047912    5393 generic.go:356] "Generic (PLEG): container finished" podID="98372fc1-b5f0-4444-a6f5-7d3a216cd88b" containerID="5e06e5ffee6ca23f9464d6df507240d6ef74861560121864798247757a7ce328" exitCode=0
Nov 03 10:55:09.048670 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.048116    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6" event={"ID":"98372fc1-b5f0-4444-a6f5-7d3a216cd88b","Type":"ContainerDied","Data":"5e06e5ffee6ca23f9464d6df507240d6ef74861560121864798247757a7ce328"}
Nov 03 10:55:09.080563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.080539    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:55:09.169124 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.169052    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-plugins-config-volume\") pod \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\" (UID: \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\") "
Nov 03 10:55:09.170305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.169464    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-plugins-config-volume" pod "98372fc1-b5f0-4444-a6f5-7d3a216cd88b" (UID: "98372fc1-b5f0-4444-a6f5-7d3a216cd88b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:09.170305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.169610    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-kube-api-access-kj2h4\" (UniqueName: \"kubernetes.io/projected/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-kube-api-access-kj2h4\") pod \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\" (UID: \"98372fc1-b5f0-4444-a6f5-7d3a216cd88b\") "
Nov 03 10:55:09.170305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.170273    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:09.172903 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.172320    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-kube-api-access-kj2h4" pod "98372fc1-b5f0-4444-a6f5-7d3a216cd88b" (UID: "98372fc1-b5f0-4444-a6f5-7d3a216cd88b"). InnerVolumeSpecName "kube-api-access-kj2h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:09.271583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.271549    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj2h4\" (UniqueName: \"kubernetes.io/projected/98372fc1-b5f0-4444-a6f5-7d3a216cd88b-kube-api-access-kj2h4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:09.854640 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.854610    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:55:09.855096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.855071    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:55:09.856279 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.856258    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:55:09.924204 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.924170    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:55:09.925242 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.925215    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:55:09.926531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:09.926512    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:55:10.057463 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:10.057430    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"
Nov 03 10:55:10.058255 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:10.057459    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6" event={"ID":"98372fc1-b5f0-4444-a6f5-7d3a216cd88b","Type":"ContainerDied","Data":"0b99beabcae029b896b2c631cdecaef487d29e2638d5230b98cb7813f7b56c4f"}
Nov 03 10:55:10.058255 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:10.057524    5393 scope.go:122] "RemoveContainer" containerID="5e06e5ffee6ca23f9464d6df507240d6ef74861560121864798247757a7ce328"
Nov 03 10:55:10.080983 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:10.080909    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"]
Nov 03 10:55:10.084198 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:10.084170    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-35afa6-75479676bc-sp6v6"]
Nov 03 10:55:10.262845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:10.262748    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98372fc1-b5f0-4444-a6f5-7d3a216cd88b" path="/var/lib/kubelet/pods/98372fc1-b5f0-4444-a6f5-7d3a216cd88b/volumes"
Nov 03 10:55:11.979951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:11.979722    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.069943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:12.069004    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded"
Nov 03 10:55:12.481041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:12.480984    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz" podUID="955a3bdd-2c14-47e9-83fd-31b30d698855" containerName="payload-processing" containerID="cri-o://e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06" gracePeriod=25
Nov 03 10:55:12.505665 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:12.505617    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.602947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:12.602025    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:12.669085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:12.669022    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6" podUID="8564fcdb-1c3a-444d-b433-4c04b0fe3e27" containerName="payload-pre-processing" containerID="cri-o://fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755" gracePeriod=25
Nov 03 10:55:13.013944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:13.000293    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:13.107105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:13.089295    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:13.134055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:13.134006    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:13.378143 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:13.378103    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:55:13.702041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:13.701946    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"]
Nov 03 10:55:13.702242 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:13.702212    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy" containerID="cri-o://0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b" gracePeriod=30
Nov 03 10:55:14.896908 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:14.896868    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:55:14.897339 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:14.896961    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:55:15.645810 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.645777    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"]
Nov 03 10:55:15.646432 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.646417    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ad9141e9-c8b0-4315-9197-235304be49c7" containerName="revoke-keys"
Nov 03 10:55:15.646491 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.646434    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9141e9-c8b0-4315-9197-235304be49c7" containerName="revoke-keys"
Nov 03 10:55:15.646491 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.646460    5393 cpu_manager.go:414] "containerMap: removing container" podUID="98372fc1-b5f0-4444-a6f5-7d3a216cd88b" containerName="payload-processing"
Nov 03 10:55:15.646491 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.646466    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98372fc1-b5f0-4444-a6f5-7d3a216cd88b" containerName="payload-processing"
Nov 03 10:55:15.646629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.646591    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="98372fc1-b5f0-4444-a6f5-7d3a216cd88b" containerName="payload-processing"
Nov 03 10:55:15.646629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.646606    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9141e9-c8b0-4315-9197-235304be49c7" containerName="revoke-keys"
Nov 03 10:55:15.651439 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.651421    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.654772 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.654749    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:55:15.655976 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.655951    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:55:15.657445 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.657424    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"]
Nov 03 10:55:15.671382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.671278    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.678143 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.678093    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.678295 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.678172    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qblhj\" (UniqueName: \"kubernetes.io/projected/5a29ff0a-28dd-44d2-8cfb-18c767be8008-kube-api-access-qblhj\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.781435 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.780010    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.781435 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.780129    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.789162 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.780177    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qblhj\" (UniqueName: \"kubernetes.io/projected/5a29ff0a-28dd-44d2-8cfb-18c767be8008-kube-api-access-qblhj\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.789776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.789737    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.802564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.802449    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qblhj\" (UniqueName: \"kubernetes.io/projected/5a29ff0a-28dd-44d2-8cfb-18c767be8008-kube-api-access-qblhj\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.803610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.803569    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e615c94-gq2gm\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:15.967886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:15.967803    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:16.250753 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:16.250733    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"]
Nov 03 10:55:16.256038 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:55:16.255803    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a29ff0a_28dd_44d2_8cfb_18c767be8008.slice/crio-8de0796170781a4db78ba28ce161f6248a29282abfc1b694a19a49a23d51ee00 WatchSource:0}: Error finding container 8de0796170781a4db78ba28ce161f6248a29282abfc1b694a19a49a23d51ee00: Status 404 returned error can't find the container with id 8de0796170781a4db78ba28ce161f6248a29282abfc1b694a19a49a23d51ee00
Nov 03 10:55:16.343678 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:16.343647    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm" event={"ID":"5a29ff0a-28dd-44d2-8cfb-18c767be8008","Type":"ContainerStarted","Data":"8de0796170781a4db78ba28ce161f6248a29282abfc1b694a19a49a23d51ee00"}
Nov 03 10:55:17.352645 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:17.352608    5393 generic.go:356] "Generic (PLEG): container finished" podID="5a29ff0a-28dd-44d2-8cfb-18c767be8008" containerID="3a2ab793137783c741d85120d1963e8a7bf34ad831eaf3e93de561922cc25a9a" exitCode=0
Nov 03 10:55:17.353067 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:17.352685    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm" event={"ID":"5a29ff0a-28dd-44d2-8cfb-18c767be8008","Type":"ContainerDied","Data":"3a2ab793137783c741d85120d1963e8a7bf34ad831eaf3e93de561922cc25a9a"}
Nov 03 10:55:17.872989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:17.872831    5393 scope.go:122] "RemoveContainer" containerID="ae45d377f6407a332cf0c90eef157fb2751b6d34e7f22b0bd1c78509d25622f7"
Nov 03 10:55:17.928670 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:17.927223    5393 scope.go:122] "RemoveContainer" containerID="6f8d8b6ae003072564ea63663f250ef32ea076649163257889d243e84025b2f0"
Nov 03 10:55:18.570716 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.570403    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:18.666241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.666150    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-xks-ca\") pod \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") "
Nov 03 10:55:18.666417 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.666299    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a29ff0a-28dd-44d2-8cfb-18c767be8008-kube-api-access-qblhj\" (UniqueName: \"kubernetes.io/projected/5a29ff0a-28dd-44d2-8cfb-18c767be8008-kube-api-access-qblhj\") pod \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") "
Nov 03 10:55:18.666487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.666474    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-openshift-ca\") pod \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\" (UID: \"5a29ff0a-28dd-44d2-8cfb-18c767be8008\") "
Nov 03 10:55:18.667307 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.667276    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-openshift-ca" pod "5a29ff0a-28dd-44d2-8cfb-18c767be8008" (UID: "5a29ff0a-28dd-44d2-8cfb-18c767be8008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:18.669099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.669079    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-xks-ca" pod "5a29ff0a-28dd-44d2-8cfb-18c767be8008" (UID: "5a29ff0a-28dd-44d2-8cfb-18c767be8008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:18.669382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.669355    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a29ff0a-28dd-44d2-8cfb-18c767be8008-kube-api-access-qblhj" pod "5a29ff0a-28dd-44d2-8cfb-18c767be8008" (UID: "5a29ff0a-28dd-44d2-8cfb-18c767be8008"). InnerVolumeSpecName "kube-api-access-qblhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:18.768793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.768761    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:18.768793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.768791    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a29ff0a-28dd-44d2-8cfb-18c767be8008-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:18.768793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.768801    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qblhj\" (UniqueName: \"kubernetes.io/projected/5a29ff0a-28dd-44d2-8cfb-18c767be8008-kube-api-access-qblhj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:18.961719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:18.961687    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:55:19.072874 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.072841    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-podinfo\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.073086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.072980    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-data\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.073086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.073041    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-token\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-token\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.073225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.073120    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-kube-api-access-ncnc4\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-kube-api-access-ncnc4\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.073225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.073169    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-credential-socket\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.073225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.073200    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istiod-ca-cert\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.073386 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.073287    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-envoy\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.073450 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.073391    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-socket\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.073502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.073459    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-certs\") pod \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\" (UID: \"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7\") "
Nov 03 10:55:19.074558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.073904    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-data" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:19.074558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.074159    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.074558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.074184    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-socket" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:19.074558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.074521    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istiod-ca-cert" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:19.074558 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.074528    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-certs" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:19.075006 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.074789    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-credential-socket" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:19.075672 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.075646    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-podinfo" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:55:19.076683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.076663    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-envoy" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:19.076770 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.076737    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-kube-api-access-ncnc4" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "kube-api-access-ncnc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:19.077208 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.077191    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-token" pod "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" (UID: "264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:19.175080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.175048    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnc4\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-kube-api-access-ncnc4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.175080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.175076    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.175080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.175086    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.175345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.175095    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.175345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.175104    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.175345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.175118    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.175345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.175128    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.175345 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.175136    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:19.414716 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.414679    5393 generic.go:356] "Generic (PLEG): container finished" podID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerID="0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b" exitCode=0
Nov 03 10:55:19.414956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.414781    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"
Nov 03 10:55:19.414956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.414811    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" event={"ID":"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7","Type":"ContainerDied","Data":"0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b"}
Nov 03 10:55:19.414956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.414868    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx" event={"ID":"264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7","Type":"ContainerDied","Data":"f3fcc224069a23344462808d77d29a827bae6d04cae0a11a10c386de57008026"}
Nov 03 10:55:19.414956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.414892    5393 scope.go:122] "RemoveContainer" containerID="0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b"
Nov 03 10:55:19.417702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.417663    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm" event={"ID":"5a29ff0a-28dd-44d2-8cfb-18c767be8008","Type":"ContainerDied","Data":"8de0796170781a4db78ba28ce161f6248a29282abfc1b694a19a49a23d51ee00"}
Nov 03 10:55:19.417817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.417729    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de0796170781a4db78ba28ce161f6248a29282abfc1b694a19a49a23d51ee00"
Nov 03 10:55:19.417817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.417800    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"
Nov 03 10:55:19.428157 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.428135    5393 scope.go:122] "RemoveContainer" containerID="0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43"
Nov 03 10:55:19.439506 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.439475    5393 scope.go:122] "RemoveContainer" containerID="0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b"
Nov 03 10:55:19.441855 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:19.439860    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b\": container with ID starting with 0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b not found: ID does not exist" containerID="0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b"
Nov 03 10:55:19.441855 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.439901    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b"} err="failed to get container status \"0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b\": rpc error: code = NotFound desc = could not find container \"0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b\": container with ID starting with 0d7494ea04075ee04953e44fcd0528906a23fec47ba085644f0b6177d6b19e5b not found: ID does not exist"
Nov 03 10:55:19.441855 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.440003    5393 scope.go:122] "RemoveContainer" containerID="0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43"
Nov 03 10:55:19.441855 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:19.440335    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43\": container with ID starting with 0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43 not found: ID does not exist" containerID="0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43"
Nov 03 10:55:19.441855 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.440359    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43"} err="failed to get container status \"0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43\": rpc error: code = NotFound desc = could not find container \"0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43\": container with ID starting with 0529d47c3b85f135493bc9aee15109f8868b5cb7b5e489f17bcccbc7ba8acc43 not found: ID does not exist"
Nov 03 10:55:19.449605 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.447739    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"]
Nov 03 10:55:19.451488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:19.451466    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-35afa6-openshift-default-6c47ff9c57-z87gx"]
Nov 03 10:55:20.221487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:20.221447    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz" podUID="955a3bdd-2c14-47e9-83fd-31b30d698855" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 10:55:20.221487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:20.221464    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6" podUID="8564fcdb-1c3a-444d-b433-4c04b0fe3e27" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 10:55:20.230899 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:20.230872    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" path="/var/lib/kubelet/pods/264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7/volumes"
Nov 03 10:55:24.333060 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:24.333026    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"]
Nov 03 10:55:24.342001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:24.341973    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"]
Nov 03 10:55:28.444414 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:28.444337    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"]
Nov 03 10:55:28.444868 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:28.444647    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy" containerID="cri-o://1866da88411e647035ffa9008bfdfe35b0e12075836f1cf684dda8bb5a722868" gracePeriod=30
Nov 03 10:55:28.730888 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:28.730795    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"]
Nov 03 10:55:28.731127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:28.731066    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2" podUID="ba214d7f-08af-498e-a14f-f3151a3423d3" containerName="payload-pre-processing" containerID="cri-o://0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520" gracePeriod=30
Nov 03 10:55:29.285410 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.276066    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:55:29.316433 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.316344    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba214d7f-08af-498e-a14f-f3151a3423d3-kube-api-access-dq257\" (UniqueName: \"kubernetes.io/projected/ba214d7f-08af-498e-a14f-f3151a3423d3-kube-api-access-dq257\") pod \"ba214d7f-08af-498e-a14f-f3151a3423d3\" (UID: \"ba214d7f-08af-498e-a14f-f3151a3423d3\") "
Nov 03 10:55:29.316730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.316707    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba214d7f-08af-498e-a14f-f3151a3423d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba214d7f-08af-498e-a14f-f3151a3423d3-plugins-config-volume\") pod \"ba214d7f-08af-498e-a14f-f3151a3423d3\" (UID: \"ba214d7f-08af-498e-a14f-f3151a3423d3\") "
Nov 03 10:55:29.323178 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.323152    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba214d7f-08af-498e-a14f-f3151a3423d3-plugins-config-volume" pod "ba214d7f-08af-498e-a14f-f3151a3423d3" (UID: "ba214d7f-08af-498e-a14f-f3151a3423d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:29.324096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.324072    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba214d7f-08af-498e-a14f-f3151a3423d3-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:29.326049 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.326020    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba214d7f-08af-498e-a14f-f3151a3423d3-kube-api-access-dq257" pod "ba214d7f-08af-498e-a14f-f3151a3423d3" (UID: "ba214d7f-08af-498e-a14f-f3151a3423d3"). InnerVolumeSpecName "kube-api-access-dq257". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:29.334301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.334264    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc" podUID="d1937d6c-7aa5-4e49-bffa-18f54cb3aada" containerName="payload-processing" containerID="cri-o://b12e465d984eb2bdbec9ace6bdef8a640bfbe8caaa000b58467144d446a27b72" gracePeriod=25
Nov 03 10:55:29.342477 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.342286    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7" podUID="3e789eb9-9cf4-42ed-81a7-dc4097993f11" containerName="payload-pre-processing" containerID="cri-o://668f7ae8096c7f19595484c57bd2772923f48f12a4254939538b748f61ca2133" gracePeriod=25
Nov 03 10:55:29.425296 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.425267    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq257\" (UniqueName: \"kubernetes.io/projected/ba214d7f-08af-498e-a14f-f3151a3423d3-kube-api-access-dq257\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:29.511984 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.511902    5393 generic.go:356] "Generic (PLEG): container finished" podID="d1937d6c-7aa5-4e49-bffa-18f54cb3aada" containerID="b12e465d984eb2bdbec9ace6bdef8a640bfbe8caaa000b58467144d446a27b72" exitCode=0
Nov 03 10:55:29.512680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.512080    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc" event={"ID":"d1937d6c-7aa5-4e49-bffa-18f54cb3aada","Type":"ContainerDied","Data":"b12e465d984eb2bdbec9ace6bdef8a640bfbe8caaa000b58467144d446a27b72"}
Nov 03 10:55:29.514546 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.514520    5393 generic.go:356] "Generic (PLEG): container finished" podID="ba214d7f-08af-498e-a14f-f3151a3423d3" containerID="0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520" exitCode=0
Nov 03 10:55:29.514654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.514632    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2" event={"ID":"ba214d7f-08af-498e-a14f-f3151a3423d3","Type":"ContainerDied","Data":"0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520"}
Nov 03 10:55:29.514713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.514673    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2" event={"ID":"ba214d7f-08af-498e-a14f-f3151a3423d3","Type":"ContainerDied","Data":"a3f3f0455000622cfb570178ad10aba172aa6488f02b7569fdf14f89941221f2"}
Nov 03 10:55:29.514713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.514700    5393 scope.go:122] "RemoveContainer" containerID="0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520"
Nov 03 10:55:29.514862 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.514847    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"
Nov 03 10:55:29.521841 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.521810    5393 generic.go:356] "Generic (PLEG): container finished" podID="3e789eb9-9cf4-42ed-81a7-dc4097993f11" containerID="668f7ae8096c7f19595484c57bd2772923f48f12a4254939538b748f61ca2133" exitCode=0
Nov 03 10:55:29.521960 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.521934    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7" event={"ID":"3e789eb9-9cf4-42ed-81a7-dc4097993f11","Type":"ContainerDied","Data":"668f7ae8096c7f19595484c57bd2772923f48f12a4254939538b748f61ca2133"}
Nov 03 10:55:29.574090 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.573089    5393 scope.go:122] "RemoveContainer" containerID="0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520"
Nov 03 10:55:29.574090 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:29.573567    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520\": container with ID starting with 0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520 not found: ID does not exist" containerID="0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520"
Nov 03 10:55:29.574090 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.573603    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520"} err="failed to get container status \"0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520\": rpc error: code = NotFound desc = could not find container \"0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520\": container with ID starting with 0ddbbf6076130dc2fe7bbf4e711e2efd949034d112b9b8f1be79365a2e636520 not found: ID does not exist"
Nov 03 10:55:29.589086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.589057    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"]
Nov 03 10:55:29.595555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.595529    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-18264e-64dc9f8f47-j7mf2"]
Nov 03 10:55:29.724005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.723971    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:55:29.775326 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.775304    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:55:29.846075 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.845993    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e789eb9-9cf4-42ed-81a7-dc4097993f11-kube-api-access-b8lzb\" (UniqueName: \"kubernetes.io/projected/3e789eb9-9cf4-42ed-81a7-dc4097993f11-kube-api-access-b8lzb\") pod \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\" (UID: \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\") "
Nov 03 10:55:29.846261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.846152    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e789eb9-9cf4-42ed-81a7-dc4097993f11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e789eb9-9cf4-42ed-81a7-dc4097993f11-plugins-config-volume\") pod \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\" (UID: \"3e789eb9-9cf4-42ed-81a7-dc4097993f11\") "
Nov 03 10:55:29.846346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.846275    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-kube-api-access-8ktbm\" (UniqueName: \"kubernetes.io/projected/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-kube-api-access-8ktbm\") pod \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\" (UID: \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\") "
Nov 03 10:55:29.846405 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.846350    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-plugins-config-volume\") pod \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\" (UID: \"d1937d6c-7aa5-4e49-bffa-18f54cb3aada\") "
Nov 03 10:55:29.846684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.846644    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e789eb9-9cf4-42ed-81a7-dc4097993f11-plugins-config-volume" pod "3e789eb9-9cf4-42ed-81a7-dc4097993f11" (UID: "3e789eb9-9cf4-42ed-81a7-dc4097993f11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:29.846837 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.846689    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-plugins-config-volume" pod "d1937d6c-7aa5-4e49-bffa-18f54cb3aada" (UID: "d1937d6c-7aa5-4e49-bffa-18f54cb3aada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:29.847029 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.847010    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e789eb9-9cf4-42ed-81a7-dc4097993f11-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:29.847105 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.847035    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:29.848574 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.848554    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e789eb9-9cf4-42ed-81a7-dc4097993f11-kube-api-access-b8lzb" pod "3e789eb9-9cf4-42ed-81a7-dc4097993f11" (UID: "3e789eb9-9cf4-42ed-81a7-dc4097993f11"). InnerVolumeSpecName "kube-api-access-b8lzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:29.848655 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.848585    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-kube-api-access-8ktbm" pod "d1937d6c-7aa5-4e49-bffa-18f54cb3aada" (UID: "d1937d6c-7aa5-4e49-bffa-18f54cb3aada"). InnerVolumeSpecName "kube-api-access-8ktbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:29.948310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.948281    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8lzb\" (UniqueName: \"kubernetes.io/projected/3e789eb9-9cf4-42ed-81a7-dc4097993f11-kube-api-access-b8lzb\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:29.948310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:29.948307    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ktbm\" (UniqueName: \"kubernetes.io/projected/d1937d6c-7aa5-4e49-bffa-18f54cb3aada-kube-api-access-8ktbm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:30.220411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.220332    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6" podUID="8564fcdb-1c3a-444d-b433-4c04b0fe3e27" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 10:55:30.220411 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.220329    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz" podUID="955a3bdd-2c14-47e9-83fd-31b30d698855" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 10:55:30.234945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.233195    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba214d7f-08af-498e-a14f-f3151a3423d3" path="/var/lib/kubelet/pods/ba214d7f-08af-498e-a14f-f3151a3423d3/volumes"
Nov 03 10:55:30.567560 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.567443    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7" event={"ID":"3e789eb9-9cf4-42ed-81a7-dc4097993f11","Type":"ContainerDied","Data":"44ab52d53c64e5ccea492dd1e503ebd701fc62e7e8b7abe89a809a6a01a3637b"}
Nov 03 10:55:30.568125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.567555    5393 scope.go:122] "RemoveContainer" containerID="668f7ae8096c7f19595484c57bd2772923f48f12a4254939538b748f61ca2133"
Nov 03 10:55:30.568125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.567716    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"
Nov 03 10:55:30.570638 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.570610    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc" event={"ID":"d1937d6c-7aa5-4e49-bffa-18f54cb3aada","Type":"ContainerDied","Data":"8f2891694ad34e7c5f4b754261ab40547cd0f59c70fcd44510cceb82e4f628ea"}
Nov 03 10:55:30.570771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.570735    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"
Nov 03 10:55:30.579701 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.579680    5393 scope.go:122] "RemoveContainer" containerID="b12e465d984eb2bdbec9ace6bdef8a640bfbe8caaa000b58467144d446a27b72"
Nov 03 10:55:30.590509 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.590486    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"]
Nov 03 10:55:30.593867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.593846    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-343e30-75cdc6b68c-nxxn7"]
Nov 03 10:55:30.606299 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.606272    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"]
Nov 03 10:55:30.612008 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:30.611986    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-343e30-56d96cd497-6rxxc"]
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.227130    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"]
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.227973    5393 cpu_manager.go:414] "containerMap: removing container" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.227994    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228015    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5a29ff0a-28dd-44d2-8cfb-18c767be8008" containerName="revoke-keys"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228023    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a29ff0a-28dd-44d2-8cfb-18c767be8008" containerName="revoke-keys"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228092    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ba214d7f-08af-498e-a14f-f3151a3423d3" containerName="payload-pre-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228102    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba214d7f-08af-498e-a14f-f3151a3423d3" containerName="payload-pre-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228121    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d1937d6c-7aa5-4e49-bffa-18f54cb3aada" containerName="payload-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228129    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1937d6c-7aa5-4e49-bffa-18f54cb3aada" containerName="payload-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228150    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3e789eb9-9cf4-42ed-81a7-dc4097993f11" containerName="payload-pre-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228158    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e789eb9-9cf4-42ed-81a7-dc4097993f11" containerName="payload-pre-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228359    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a29ff0a-28dd-44d2-8cfb-18c767be8008" containerName="revoke-keys"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228386    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba214d7f-08af-498e-a14f-f3151a3423d3" containerName="payload-pre-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228395    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1937d6c-7aa5-4e49-bffa-18f54cb3aada" containerName="payload-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228406    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e789eb9-9cf4-42ed-81a7-dc4097993f11" containerName="payload-pre-processing"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228415    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy"
Nov 03 10:55:31.243978 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.228425    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy"
Nov 03 10:55:31.252363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.252337    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.254587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.254560    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"]
Nov 03 10:55:31.256020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.255998    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:55:31.257350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.257324    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 10:55:31.376640 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.376601    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44xr\" (UniqueName: \"kubernetes.io/projected/071535ef-1acc-4c69-ba0c-0d4785cb97b1-kube-api-access-p44xr\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.376812 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.376664    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.377028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.377003    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.478702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.478658    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.478907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.478746    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44xr\" (UniqueName: \"kubernetes.io/projected/071535ef-1acc-4c69-ba0c-0d4785cb97b1-kube-api-access-p44xr\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.478907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.478774    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.479401 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.479377    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.481270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.481251    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.486782 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.486756    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44xr\" (UniqueName: \"kubernetes.io/projected/071535ef-1acc-4c69-ba0c-0d4785cb97b1-kube-api-access-p44xr\") pod \"maas-api-revoke-sub-e0080ac5-kx82p\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.569562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.569486    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:31.858991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:31.858955    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"]
Nov 03 10:55:31.866208 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:55:31.866176    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071535ef_1acc_4c69_ba0c_0d4785cb97b1.slice/crio-4d9f1ae3404be5621fbfe871ed7c45df0d226661448b8b43633e06b863ac4404 WatchSource:0}: Error finding container 4d9f1ae3404be5621fbfe871ed7c45df0d226661448b8b43633e06b863ac4404: Status 404 returned error can't find the container with id 4d9f1ae3404be5621fbfe871ed7c45df0d226661448b8b43633e06b863ac4404
Nov 03 10:55:32.408445 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:32.407855    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e789eb9-9cf4-42ed-81a7-dc4097993f11" path="/var/lib/kubelet/pods/3e789eb9-9cf4-42ed-81a7-dc4097993f11/volumes"
Nov 03 10:55:32.421764 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:32.409524    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1937d6c-7aa5-4e49-bffa-18f54cb3aada" path="/var/lib/kubelet/pods/d1937d6c-7aa5-4e49-bffa-18f54cb3aada/volumes"
Nov 03 10:55:32.593992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:32.593959    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p" event={"ID":"071535ef-1acc-4c69-ba0c-0d4785cb97b1","Type":"ContainerStarted","Data":"6bff190cb85973accb1e3e2e6bffff99991080c08429932b89e1b054b1458c3c"}
Nov 03 10:55:32.594386 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:32.594007    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p" event={"ID":"071535ef-1acc-4c69-ba0c-0d4785cb97b1","Type":"ContainerStarted","Data":"4d9f1ae3404be5621fbfe871ed7c45df0d226661448b8b43633e06b863ac4404"}
Nov 03 10:55:32.610619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:32.610563    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p" podStartSLOduration=1.610546757 podStartE2EDuration="1.610546757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:55:32.608476442 +0000 UTC m=+1576.513330882" watchObservedRunningTime="2026-11-03 10:55:32.610546757 +0000 UTC m=+1576.515401198"
Nov 03 10:55:32.922483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:32.922178    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:55:33.106009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.105554    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955a3bdd-2c14-47e9-83fd-31b30d698855-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955a3bdd-2c14-47e9-83fd-31b30d698855-plugins-config-volume\") pod \"955a3bdd-2c14-47e9-83fd-31b30d698855\" (UID: \"955a3bdd-2c14-47e9-83fd-31b30d698855\") "
Nov 03 10:55:33.106009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.105688    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955a3bdd-2c14-47e9-83fd-31b30d698855-kube-api-access-pt544\" (UniqueName: \"kubernetes.io/projected/955a3bdd-2c14-47e9-83fd-31b30d698855-kube-api-access-pt544\") pod \"955a3bdd-2c14-47e9-83fd-31b30d698855\" (UID: \"955a3bdd-2c14-47e9-83fd-31b30d698855\") "
Nov 03 10:55:33.110076 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.110028    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955a3bdd-2c14-47e9-83fd-31b30d698855-kube-api-access-pt544" pod "955a3bdd-2c14-47e9-83fd-31b30d698855" (UID: "955a3bdd-2c14-47e9-83fd-31b30d698855"). InnerVolumeSpecName "kube-api-access-pt544". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:33.110433 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.110407    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955a3bdd-2c14-47e9-83fd-31b30d698855-plugins-config-volume" pod "955a3bdd-2c14-47e9-83fd-31b30d698855" (UID: "955a3bdd-2c14-47e9-83fd-31b30d698855"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:33.208958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.208869    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/955a3bdd-2c14-47e9-83fd-31b30d698855-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.208958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.208929    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt544\" (UniqueName: \"kubernetes.io/projected/955a3bdd-2c14-47e9-83fd-31b30d698855-kube-api-access-pt544\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.257746 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.257714    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:55:33.310457 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.310420    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-kube-api-access-h9bz6\" (UniqueName: \"kubernetes.io/projected/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-kube-api-access-h9bz6\") pod \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\" (UID: \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\") "
Nov 03 10:55:33.310633 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.310522    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-plugins-config-volume\") pod \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\" (UID: \"8564fcdb-1c3a-444d-b433-4c04b0fe3e27\") "
Nov 03 10:55:33.311662 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.311633    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-plugins-config-volume" pod "8564fcdb-1c3a-444d-b433-4c04b0fe3e27" (UID: "8564fcdb-1c3a-444d-b433-4c04b0fe3e27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:33.321505 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.321480    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-kube-api-access-h9bz6" pod "8564fcdb-1c3a-444d-b433-4c04b0fe3e27" (UID: "8564fcdb-1c3a-444d-b433-4c04b0fe3e27"). InnerVolumeSpecName "kube-api-access-h9bz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:33.412944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.412842    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9bz6\" (UniqueName: \"kubernetes.io/projected/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-kube-api-access-h9bz6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.412944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.412880    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8564fcdb-1c3a-444d-b433-4c04b0fe3e27-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.603318 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.603274    5393 generic.go:356] "Generic (PLEG): container finished" podID="8564fcdb-1c3a-444d-b433-4c04b0fe3e27" containerID="fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755" exitCode=0
Nov 03 10:55:33.604005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.603336    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"
Nov 03 10:55:33.604005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.603350    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6" event={"ID":"8564fcdb-1c3a-444d-b433-4c04b0fe3e27","Type":"ContainerDied","Data":"fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755"}
Nov 03 10:55:33.604005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.603437    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6" event={"ID":"8564fcdb-1c3a-444d-b433-4c04b0fe3e27","Type":"ContainerDied","Data":"45f4dad494a2d773eb447138cd6d97d6256aaa514f14c15b6989ba79bf1d0fe8"}
Nov 03 10:55:33.604005 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.603464    5393 scope.go:122] "RemoveContainer" containerID="fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755"
Nov 03 10:55:33.605596 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.605571    5393 generic.go:356] "Generic (PLEG): container finished" podID="071535ef-1acc-4c69-ba0c-0d4785cb97b1" containerID="6bff190cb85973accb1e3e2e6bffff99991080c08429932b89e1b054b1458c3c" exitCode=0
Nov 03 10:55:33.605771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.605732    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p" event={"ID":"071535ef-1acc-4c69-ba0c-0d4785cb97b1","Type":"ContainerDied","Data":"6bff190cb85973accb1e3e2e6bffff99991080c08429932b89e1b054b1458c3c"}
Nov 03 10:55:33.608691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.608668    5393 generic.go:356] "Generic (PLEG): container finished" podID="37c1b717-daba-4377-ad8d-5cc357d45428" containerID="1866da88411e647035ffa9008bfdfe35b0e12075836f1cf684dda8bb5a722868" exitCode=0
Nov 03 10:55:33.608799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.608760    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" event={"ID":"37c1b717-daba-4377-ad8d-5cc357d45428","Type":"ContainerDied","Data":"1866da88411e647035ffa9008bfdfe35b0e12075836f1cf684dda8bb5a722868"}
Nov 03 10:55:33.610659 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.610627    5393 generic.go:356] "Generic (PLEG): container finished" podID="955a3bdd-2c14-47e9-83fd-31b30d698855" containerID="e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06" exitCode=0
Nov 03 10:55:33.610823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.610748    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"
Nov 03 10:55:33.611124 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.610759    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz" event={"ID":"955a3bdd-2c14-47e9-83fd-31b30d698855","Type":"ContainerDied","Data":"e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06"}
Nov 03 10:55:33.611124 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.611089    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz" event={"ID":"955a3bdd-2c14-47e9-83fd-31b30d698855","Type":"ContainerDied","Data":"30bf52acd338ec5b07d961e452fac53bf9a683f418f1f8e8db5c9a2fbb29f82b"}
Nov 03 10:55:33.617176 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.617136    5393 scope.go:122] "RemoveContainer" containerID="fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755"
Nov 03 10:55:33.617547 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:33.617512    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755\": container with ID starting with fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755 not found: ID does not exist" containerID="fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755"
Nov 03 10:55:33.617650 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.617557    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755"} err="failed to get container status \"fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755\": rpc error: code = NotFound desc = could not find container \"fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755\": container with ID starting with fc7f53747e8edcfce915ec2a5f14dd35a97d4809bb9217c887da7837d1500755 not found: ID does not exist"
Nov 03 10:55:33.617650 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.617575    5393 scope.go:122] "RemoveContainer" containerID="e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06"
Nov 03 10:55:33.628898 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.628721    5393 scope.go:122] "RemoveContainer" containerID="e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06"
Nov 03 10:55:33.630163 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:33.630092    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06\": container with ID starting with e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06 not found: ID does not exist" containerID="e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06"
Nov 03 10:55:33.630163 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.630135    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06"} err="failed to get container status \"e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06\": rpc error: code = NotFound desc = could not find container \"e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06\": container with ID starting with e6746b23f4d7288773fb6f79128c66bce3be0787b76b5e2e811536ef4c366f06 not found: ID does not exist"
Nov 03 10:55:33.641075 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.641049    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"]
Nov 03 10:55:33.652095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.652067    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cd6d01-778dc7f855-dgvb6"]
Nov 03 10:55:33.736003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.729843    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"]
Nov 03 10:55:33.737935 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.737886    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cd6d01-769f858f87-x4psz"]
Nov 03 10:55:33.797282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.797257    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:55:33.954253 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954221    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-certs\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.954451 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954271    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c1b717-daba-4377-ad8d-5cc357d45428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c1b717-daba-4377-ad8d-5cc357d45428-istiod-ca-cert\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.954451 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954402    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-istio-token\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-istio-token\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.954579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954458    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-credential-socket\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.954579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954502    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-envoy\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.954579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954522    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-certs" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:33.954579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954536    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-data\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.954829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954629    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-kube-api-access-5dngg\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-kube-api-access-5dngg\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.954829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954663    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37c1b717-daba-4377-ad8d-5cc357d45428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c1b717-daba-4377-ad8d-5cc357d45428-istio-podinfo\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.954829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954709    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-credential-socket" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:33.954829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954739    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c1b717-daba-4377-ad8d-5cc357d45428-istiod-ca-cert" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:33.954829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.954787    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-socket\") pod \"37c1b717-daba-4377-ad8d-5cc357d45428\" (UID: \"37c1b717-daba-4377-ad8d-5cc357d45428\") "
Nov 03 10:55:33.955237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.955018    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-data" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:33.955237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.955185    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-socket" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:33.955479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.955427    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.955479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.955457    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.955479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.955471    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c1b717-daba-4377-ad8d-5cc357d45428-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.955695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.955485    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.955695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.955499    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:33.957549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.957525    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-istio-token" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:33.957632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.957544    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-kube-api-access-5dngg" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "kube-api-access-5dngg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:33.957879 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.957860    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-envoy" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:33.958127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:33.958108    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37c1b717-daba-4377-ad8d-5cc357d45428-istio-podinfo" pod "37c1b717-daba-4377-ad8d-5cc357d45428" (UID: "37c1b717-daba-4377-ad8d-5cc357d45428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:55:34.056474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.056394    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:34.056474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.056420    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c1b717-daba-4377-ad8d-5cc357d45428-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:34.056474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.056430    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dngg\" (UniqueName: \"kubernetes.io/projected/37c1b717-daba-4377-ad8d-5cc357d45428-kube-api-access-5dngg\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:34.056474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.056440    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c1b717-daba-4377-ad8d-5cc357d45428-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:34.230755 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.230719    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8564fcdb-1c3a-444d-b433-4c04b0fe3e27" path="/var/lib/kubelet/pods/8564fcdb-1c3a-444d-b433-4c04b0fe3e27/volumes"
Nov 03 10:55:34.231222 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.231206    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955a3bdd-2c14-47e9-83fd-31b30d698855" path="/var/lib/kubelet/pods/955a3bdd-2c14-47e9-83fd-31b30d698855/volumes"
Nov 03 10:55:34.618985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.618950    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2" event={"ID":"37c1b717-daba-4377-ad8d-5cc357d45428","Type":"ContainerDied","Data":"f715baac9e08a5a19e6a937bb58829ae731dbd2a46a28bed21eaf6e9effa8c96"}
Nov 03 10:55:34.619476 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.619022    5393 scope.go:122] "RemoveContainer" containerID="1866da88411e647035ffa9008bfdfe35b0e12075836f1cf684dda8bb5a722868"
Nov 03 10:55:34.619476 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.618959    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"
Nov 03 10:55:34.641488 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.641462    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"]
Nov 03 10:55:34.644629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.644594    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-n9mq2"]
Nov 03 10:55:34.774765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.774741    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:34.868936 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.868894    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071535ef-1acc-4c69-ba0c-0d4785cb97b1-kube-api-access-p44xr\" (UniqueName: \"kubernetes.io/projected/071535ef-1acc-4c69-ba0c-0d4785cb97b1-kube-api-access-p44xr\") pod \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") "
Nov 03 10:55:34.869158 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.868974    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-xks-ca\") pod \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") "
Nov 03 10:55:34.869158 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.869027    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-openshift-ca\") pod \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\" (UID: \"071535ef-1acc-4c69-ba0c-0d4785cb97b1\") "
Nov 03 10:55:34.869415 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.869394    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-openshift-ca" pod "071535ef-1acc-4c69-ba0c-0d4785cb97b1" (UID: "071535ef-1acc-4c69-ba0c-0d4785cb97b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:34.869563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.869546    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:34.871131 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.871107    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071535ef-1acc-4c69-ba0c-0d4785cb97b1-kube-api-access-p44xr" pod "071535ef-1acc-4c69-ba0c-0d4785cb97b1" (UID: "071535ef-1acc-4c69-ba0c-0d4785cb97b1"). InnerVolumeSpecName "kube-api-access-p44xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:34.871317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.871300    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-xks-ca" pod "071535ef-1acc-4c69-ba0c-0d4785cb97b1" (UID: "071535ef-1acc-4c69-ba0c-0d4785cb97b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:55:34.970528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.970498    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44xr\" (UniqueName: \"kubernetes.io/projected/071535ef-1acc-4c69-ba0c-0d4785cb97b1-kube-api-access-p44xr\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:34.970528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:34.970526    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071535ef-1acc-4c69-ba0c-0d4785cb97b1-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:35.629235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:35.629212    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"
Nov 03 10:55:35.629832 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:35.629202    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p" event={"ID":"071535ef-1acc-4c69-ba0c-0d4785cb97b1","Type":"ContainerDied","Data":"4d9f1ae3404be5621fbfe871ed7c45df0d226661448b8b43633e06b863ac4404"}
Nov 03 10:55:35.629832 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:35.629327    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d9f1ae3404be5621fbfe871ed7c45df0d226661448b8b43633e06b863ac4404"
Nov 03 10:55:36.231601 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:36.231569    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" path="/var/lib/kubelet/pods/37c1b717-daba-4377-ad8d-5cc357d45428/volumes"
Nov 03 10:55:36.361082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:36.361050    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"]
Nov 03 10:55:36.361325 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:36.361299    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy" containerID="cri-o://6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42" gracePeriod=30
Nov 03 10:55:38.734547 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:38.734510    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"]
Nov 03 10:55:38.735027 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:38.734811    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" containerID="cri-o://2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e" gracePeriod=30
Nov 03 10:55:39.970683 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:39.970648    5393 patch_prober.go:29] interesting pod/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:55:39.971434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:39.971404    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:55:41.672702 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.672679    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:55:41.685482 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.685451    5393 generic.go:356] "Generic (PLEG): container finished" podID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerID="6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42" exitCode=0
Nov 03 10:55:41.685660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.685508    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"
Nov 03 10:55:41.685660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.685534    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" event={"ID":"48beb4a5-a3c0-46b3-916c-f7685067032b","Type":"ContainerDied","Data":"6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42"}
Nov 03 10:55:41.685660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.685589    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv" event={"ID":"48beb4a5-a3c0-46b3-916c-f7685067032b","Type":"ContainerDied","Data":"d031756a5dfcde9b6e395f42454ca07f80cab5f2911e74a1965c8116800273d4"}
Nov 03 10:55:41.685660 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.685611    5393 scope.go:122] "RemoveContainer" containerID="6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42"
Nov 03 10:55:41.703229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.703210    5393 scope.go:122] "RemoveContainer" containerID="1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550"
Nov 03 10:55:41.718584 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.718563    5393 scope.go:122] "RemoveContainer" containerID="6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42"
Nov 03 10:55:41.718994 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:41.718943    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42\": container with ID starting with 6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42 not found: ID does not exist" containerID="6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42"
Nov 03 10:55:41.719100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.719003    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42"} err="failed to get container status \"6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42\": rpc error: code = NotFound desc = could not find container \"6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42\": container with ID starting with 6ececce9060a2d1b3c6aa27b0084430adfdf012ec8cd6576c2b096a2837cbb42 not found: ID does not exist"
Nov 03 10:55:41.719100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.719031    5393 scope.go:122] "RemoveContainer" containerID="1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550"
Nov 03 10:55:41.719491 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:41.719458    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550\": container with ID starting with 1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550 not found: ID does not exist" containerID="1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550"
Nov 03 10:55:41.719570 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.719489    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550"} err="failed to get container status \"1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550\": rpc error: code = NotFound desc = could not find container \"1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550\": container with ID starting with 1a9bbb6e92056cc0c3a7f73dc4efaadb5a51f510cdf488f92a1a7a021b2c0550 not found: ID does not exist"
Nov 03 10:55:41.809118 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809088    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-credential-socket\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809223    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48beb4a5-a3c0-46b3-916c-f7685067032b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48beb4a5-a3c0-46b3-916c-f7685067032b-istiod-ca-cert\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809271    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-kube-api-access-44bvs\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-kube-api-access-44bvs\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809449 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809352    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-socket\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809449 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809419    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-certs\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809449 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809350    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-credential-socket" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:41.809603 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809453    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-token\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-token\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809603 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809533    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-socket" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:41.809603 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809541    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-envoy\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809603 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809578    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-podinfo\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809638    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-data\") pod \"48beb4a5-a3c0-46b3-916c-f7685067032b\" (UID: \"48beb4a5-a3c0-46b3-916c-f7685067032b\") "
Nov 03 10:55:41.809817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809642    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-certs" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:41.809817 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809650    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48beb4a5-a3c0-46b3-916c-f7685067032b-istiod-ca-cert" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:41.810042 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.809965    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-data" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:41.810382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.810310    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48beb4a5-a3c0-46b3-916c-f7685067032b-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:41.810382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.810334    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:41.810382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.810349    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:41.810382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.810357    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:41.810382 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.810366    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:41.811802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.811777    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-podinfo" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:55:41.811910 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.811842    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-envoy" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:41.812156 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.812135    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-token" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:41.812253 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.812162    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-kube-api-access-44bvs" pod "48beb4a5-a3c0-46b3-916c-f7685067032b" (UID: "48beb4a5-a3c0-46b3-916c-f7685067032b"). InnerVolumeSpecName "kube-api-access-44bvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:41.911932 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.911832    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bvs\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-kube-api-access-44bvs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:41.911932 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.911863    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:41.911932 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.911873    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:41.911932 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:41.911882    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48beb4a5-a3c0-46b3-916c-f7685067032b-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:42.014869 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:42.014835    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"]
Nov 03 10:55:42.018562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:42.018534    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-18264e-openshift-default-69d4b49699-7hrmv"]
Nov 03 10:55:42.231337 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:42.231263    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" path="/var/lib/kubelet/pods/48beb4a5-a3c0-46b3-916c-f7685067032b/volumes"
Nov 03 10:55:43.190059 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:43.190026    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:55:44.009095 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.009032    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:55:44.152845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.152811    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-podinfo\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.152862    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-token\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-token\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.152952    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-data\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153006    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-credential-socket\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153025    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-envoy\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153080    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-socket\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153428 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153130    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-certs\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153428 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153154    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istiod-ca-cert\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153428 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153201    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-kube-api-access-ksd5s\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-kube-api-access-ksd5s\") pod \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\" (UID: \"068911eb-8f9c-4e54-a792-ee9a96ff8cf7\") "
Nov 03 10:55:44.153612 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153587    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-data" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:44.153948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153859    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.154040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.153979    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-socket" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:44.154110 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.154082    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-certs" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:44.154286 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.154180    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-credential-socket" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:44.154405 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.154306    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istiod-ca-cert" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:55:44.156007 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.155981    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-token" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:44.156131 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.156046    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-podinfo" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:55:44.156216 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.156200    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-kube-api-access-ksd5s" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "kube-api-access-ksd5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:55:44.156281 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.156266    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-envoy" pod "068911eb-8f9c-4e54-a792-ee9a96ff8cf7" (UID: "068911eb-8f9c-4e54-a792-ee9a96ff8cf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:55:44.258990 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.258710    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.258990 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.258749    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.258990 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.258763    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.258990 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.258780    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksd5s\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-kube-api-access-ksd5s\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.258990 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.258796    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.258990 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.258809    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.258990 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.258822    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.258990 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.258835    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068911eb-8f9c-4e54-a792-ee9a96ff8cf7-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:55:44.708057 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.708022    5393 generic.go:356] "Generic (PLEG): container finished" podID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerID="2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e" exitCode=0
Nov 03 10:55:44.708233 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.708071    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"
Nov 03 10:55:44.708233 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.708102    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" event={"ID":"068911eb-8f9c-4e54-a792-ee9a96ff8cf7","Type":"ContainerDied","Data":"2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e"}
Nov 03 10:55:44.708233 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.708156    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl" event={"ID":"068911eb-8f9c-4e54-a792-ee9a96ff8cf7","Type":"ContainerDied","Data":"f49f0fc2cdde18a44f99f8da961cd6f5535a3478c275ef511d861d10b763d1b3"}
Nov 03 10:55:44.708233 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.708179    5393 scope.go:122] "RemoveContainer" containerID="2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e"
Nov 03 10:55:44.718303 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.718286    5393 scope.go:122] "RemoveContainer" containerID="1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4"
Nov 03 10:55:44.728576 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.728553    5393 scope.go:122] "RemoveContainer" containerID="2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e"
Nov 03 10:55:44.728720 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.728630    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"]
Nov 03 10:55:44.728958 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:44.728938    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e\": container with ID starting with 2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e not found: ID does not exist" containerID="2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e"
Nov 03 10:55:44.729041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.728970    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e"} err="failed to get container status \"2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e\": rpc error: code = NotFound desc = could not find container \"2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e\": container with ID starting with 2e699545f2523144c265c9bc408fdc63cee29a782806c436fd8d00b45f37190e not found: ID does not exist"
Nov 03 10:55:44.729041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.728995    5393 scope.go:122] "RemoveContainer" containerID="1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4"
Nov 03 10:55:44.729351 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:55:44.729323    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4\": container with ID starting with 1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4 not found: ID does not exist" containerID="1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4"
Nov 03 10:55:44.729399 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.729359    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4"} err="failed to get container status \"1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4\": rpc error: code = NotFound desc = could not find container \"1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4\": container with ID starting with 1c023d71b6f693994823918f796cfaab22649edf23873404054f792ce3049ae4 not found: ID does not exist"
Nov 03 10:55:44.731693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:44.731673    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-343e30-openshift-default-6ffd77848f-4bxxl"]
Nov 03 10:55:46.245049 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:46.245014    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" path="/var/lib/kubelet/pods/068911eb-8f9c-4e54-a792-ee9a96ff8cf7/volumes"
Nov 03 10:55:48.479943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:48.479420    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.563082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:48.563036    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.949381 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:48.949331    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 10:55:48.950040 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:48.950004    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:48.962948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:48.950202    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.098079 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:49.098022    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.228449 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:49.228348    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:55:49.240393 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:49.240348    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.571121 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:49.571006    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.582427 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:49.578766    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:55:49.626418 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:49.626375    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:55:49.633853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:49.633825    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:55:56.743886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:56.743853    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"]
Nov 03 10:55:56.744618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:55:56.744578    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy" containerID="cri-o://840440d2d455cfa3e949593c0df6a1231615bad4beabf4f4ef7b37f694944de9" gracePeriod=30
Nov 03 10:56:01.984251 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:01.984193    5393 generic.go:356] "Generic (PLEG): container finished" podID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerID="840440d2d455cfa3e949593c0df6a1231615bad4beabf4f4ef7b37f694944de9" exitCode=0
Nov 03 10:56:01.985228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:01.984392    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" event={"ID":"821a9850-2564-4179-8aeb-d80fb0e4aaf0","Type":"ContainerDied","Data":"840440d2d455cfa3e949593c0df6a1231615bad4beabf4f4ef7b37f694944de9"}
Nov 03 10:56:01.985228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:01.984454    5393 scope.go:122] "RemoveContainer" containerID="f2cda572e13813b7d379c79c5e0eb4503c1e6f76a38d9d2a300580565bd32753"
Nov 03 10:56:02.029548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.029487    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:56:02.103321 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103283    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-certs\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.103534 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103483    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-socket\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.103610 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103598    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-kube-api-access-cv5mq\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-kube-api-access-cv5mq\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.103676 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103612    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-certs" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:02.103676 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103647    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-data\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.103788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103683    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-podinfo\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.103788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103729    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istiod-ca-cert\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.103788 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103771    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-token\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-token\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.104000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103843    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-credential-socket\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.104000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103880    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-envoy\") pod \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\" (UID: \"821a9850-2564-4179-8aeb-d80fb0e4aaf0\") "
Nov 03 10:56:02.104000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103767    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-socket" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:02.104000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.103981    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-data" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:02.104227 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.104088    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istiod-ca-cert" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:02.104527 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.104476    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.104527 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.104501    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.104527 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.104513    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.104527 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.104522    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.104527 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.104475    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-credential-socket" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:02.106398 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.106370    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-kube-api-access-cv5mq" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "kube-api-access-cv5mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:02.106523 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.106408    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-envoy" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:02.106617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.106596    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-token" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:02.106820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.106802    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-podinfo" pod "821a9850-2564-4179-8aeb-d80fb0e4aaf0" (UID: "821a9850-2564-4179-8aeb-d80fb0e4aaf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:56:02.205766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.205726    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv5mq\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-kube-api-access-cv5mq\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.205766 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.205765    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.206016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.205776    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.206016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.205787    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.206016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.205795    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821a9850-2564-4179-8aeb-d80fb0e4aaf0-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:02.997785 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.997747    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq" event={"ID":"821a9850-2564-4179-8aeb-d80fb0e4aaf0","Type":"ContainerDied","Data":"e527cfec27a0389ef4fe9bdc63b0d33a889ee0d177f405bc9e075b7dc7c8b240"}
Nov 03 10:56:02.997785 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.997781    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"
Nov 03 10:56:02.998336 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:02.997814    5393 scope.go:122] "RemoveContainer" containerID="840440d2d455cfa3e949593c0df6a1231615bad4beabf4f4ef7b37f694944de9"
Nov 03 10:56:03.017495 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:03.017465    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"]
Nov 03 10:56:03.024087 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:03.024062    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8bbb6e-openshift-default-669db9d899-5mggq"]
Nov 03 10:56:04.009143 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009115    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"]
Nov 03 10:56:04.009630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009617    5393 cpu_manager.go:414] "containerMap: removing container" podUID="071535ef-1acc-4c69-ba0c-0d4785cb97b1" containerName="revoke-keys"
Nov 03 10:56:04.009688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009631    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071535ef-1acc-4c69-ba0c-0d4785cb97b1" containerName="revoke-keys"
Nov 03 10:56:04.009688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009644    5393 cpu_manager.go:414] "containerMap: removing container" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy"
Nov 03 10:56:04.009688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009649    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy"
Nov 03 10:56:04.009688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009659    5393 cpu_manager.go:414] "containerMap: removing container" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy"
Nov 03 10:56:04.009688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009665    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264e0847-a3ce-4bb2-ad7f-4a69a5b03bb7" containerName="istio-proxy"
Nov 03 10:56:04.009688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009687    5393 cpu_manager.go:414] "containerMap: removing container" podUID="8564fcdb-1c3a-444d-b433-4c04b0fe3e27" containerName="payload-pre-processing"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009692    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8564fcdb-1c3a-444d-b433-4c04b0fe3e27" containerName="payload-pre-processing"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009699    5393 cpu_manager.go:414] "containerMap: removing container" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009703    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009718    5393 cpu_manager.go:414] "containerMap: removing container" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009723    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009735    5393 cpu_manager.go:414] "containerMap: removing container" podUID="955a3bdd-2c14-47e9-83fd-31b30d698855" containerName="payload-processing"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009739    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955a3bdd-2c14-47e9-83fd-31b30d698855" containerName="payload-processing"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009748    5393 cpu_manager.go:414] "containerMap: removing container" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009761    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009770    5393 cpu_manager.go:414] "containerMap: removing container" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009774    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009782    5393 cpu_manager.go:414] "containerMap: removing container" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009786    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009856    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009865    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009871    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="071535ef-1acc-4c69-ba0c-0d4785cb97b1" containerName="revoke-keys"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009879    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c1b717-daba-4377-ad8d-5cc357d45428" containerName="istio-proxy"
Nov 03 10:56:04.009882 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009889    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8564fcdb-1c3a-444d-b433-4c04b0fe3e27" containerName="payload-pre-processing"
Nov 03 10:56:04.010422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009896    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy"
Nov 03 10:56:04.010422 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.009904    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="955a3bdd-2c14-47e9-83fd-31b30d698855" containerName="payload-processing"
Nov 03 10:56:04.014708 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.014687    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:04.020671 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.020353    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"]
Nov 03 10:56:04.020671 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.020587    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-e92a08-dockercfg-g99kc\""
Nov 03 10:56:04.020671 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.020608    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-e92a08\""
Nov 03 10:56:04.054112 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.054076    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"]
Nov 03 10:56:04.054933 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.054894    5393 cpu_manager.go:414] "containerMap: removing container" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy"
Nov 03 10:56:04.054989 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.054943    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy"
Nov 03 10:56:04.055139 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.055126    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" containerName="istio-proxy"
Nov 03 10:56:04.055172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.055147    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="068911eb-8f9c-4e54-a792-ee9a96ff8cf7" containerName="istio-proxy"
Nov 03 10:56:04.055172 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.055166    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="48beb4a5-a3c0-46b3-916c-f7685067032b" containerName="istio-proxy"
Nov 03 10:56:04.062948 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.059219    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:04.064142 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.064112    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"]
Nov 03 10:56:04.145240 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.145207    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz\" (UID: \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:04.145398 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.145329    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984c6\" (UniqueName: \"kubernetes.io/projected/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-kube-api-access-984c6\") pod \"payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz\" (UID: \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:04.145455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.145404    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffd7c61-7685-449c-84d1-045d19c6148d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm\" (UID: \"2ffd7c61-7685-449c-84d1-045d19c6148d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:04.145455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.145430    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9w2\" (UniqueName: \"kubernetes.io/projected/2ffd7c61-7685-449c-84d1-045d19c6148d-kube-api-access-2f9w2\") pod \"payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm\" (UID: \"2ffd7c61-7685-449c-84d1-045d19c6148d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:04.254944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.250843    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz\" (UID: \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:04.254944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.250989    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984c6\" (UniqueName: \"kubernetes.io/projected/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-kube-api-access-984c6\") pod \"payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz\" (UID: \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:04.254944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.251093    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffd7c61-7685-449c-84d1-045d19c6148d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm\" (UID: \"2ffd7c61-7685-449c-84d1-045d19c6148d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:04.254944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.251123    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9w2\" (UniqueName: \"kubernetes.io/projected/2ffd7c61-7685-449c-84d1-045d19c6148d-kube-api-access-2f9w2\") pod \"payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm\" (UID: \"2ffd7c61-7685-449c-84d1-045d19c6148d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:04.254944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.252374    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz\" (UID: \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:04.254944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.253341    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffd7c61-7685-449c-84d1-045d19c6148d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm\" (UID: \"2ffd7c61-7685-449c-84d1-045d19c6148d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:04.286379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.285812    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9w2\" (UniqueName: \"kubernetes.io/projected/2ffd7c61-7685-449c-84d1-045d19c6148d-kube-api-access-2f9w2\") pod \"payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm\" (UID: \"2ffd7c61-7685-449c-84d1-045d19c6148d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:04.314974 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.312323    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821a9850-2564-4179-8aeb-d80fb0e4aaf0" path="/var/lib/kubelet/pods/821a9850-2564-4179-8aeb-d80fb0e4aaf0/volumes"
Nov 03 10:56:04.336767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.334840    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984c6\" (UniqueName: \"kubernetes.io/projected/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-kube-api-access-984c6\") pod \"payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz\" (UID: \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:04.336767 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.335180    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:04.402789 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.384531    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:04.691543 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.691480    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"]
Nov 03 10:56:04.692139 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:04.692116    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffd7c61_7685_449c_84d1_045d19c6148d.slice/crio-d7d6155810983e91726d6d3fafa150bdc8ad19a7c67fcf03ccb92b59e98c0134 WatchSource:0}: Error finding container d7d6155810983e91726d6d3fafa150bdc8ad19a7c67fcf03ccb92b59e98c0134: Status 404 returned error can't find the container with id d7d6155810983e91726d6d3fafa150bdc8ad19a7c67fcf03ccb92b59e98c0134
Nov 03 10:56:04.699688 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:04.699645    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"]
Nov 03 10:56:05.077992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:05.075195    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz" event={"ID":"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899","Type":"ContainerStarted","Data":"16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923"}
Nov 03 10:56:05.077992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:05.075266    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz" event={"ID":"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899","Type":"ContainerStarted","Data":"5dc97fa03bd2373c05e000729650f1aa9fde579fa935aef604fe3fbbc9fb0373"}
Nov 03 10:56:05.077992 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:05.076387    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:05.089490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:05.089447    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm" event={"ID":"2ffd7c61-7685-449c-84d1-045d19c6148d","Type":"ContainerStarted","Data":"de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0"}
Nov 03 10:56:05.089829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:05.089794    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm" event={"ID":"2ffd7c61-7685-449c-84d1-045d19c6148d","Type":"ContainerStarted","Data":"d7d6155810983e91726d6d3fafa150bdc8ad19a7c67fcf03ccb92b59e98c0134"}
Nov 03 10:56:05.090004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:05.089989    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:05.117500 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:05.117429    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm" podStartSLOduration=2.117411115 podStartE2EDuration="2.117411115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:05.116384601 +0000 UTC m=+1609.021239046" watchObservedRunningTime="2026-11-03 10:56:05.117411115 +0000 UTC m=+1609.022265556"
Nov 03 10:56:05.120166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:05.120125    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz" podStartSLOduration=1.120112732 podStartE2EDuration="1.120112732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:05.100662938 +0000 UTC m=+1609.005517379" watchObservedRunningTime="2026-11-03 10:56:05.120112732 +0000 UTC m=+1609.024967218"
Nov 03 10:56:08.953559 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:08.953515    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"]
Nov 03 10:56:08.959232 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:08.959206    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:08.975790 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:08.974967    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"]
Nov 03 10:56:08.975790 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:08.975276    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-6012b5-openshift-default-dockercfg-vv5nx\""
Nov 03 10:56:09.033854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.033818    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4113087-b863-4c49-9942-bfbbc4115fb8-istiod-ca-cert\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.033854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.033855    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjl7\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-kube-api-access-2sjl7\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.034123 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.034099    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-podinfo\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.034174 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.034150    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-certs\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.034314 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.034278    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-socket\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.034456 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.034338    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-envoy\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.034456 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.034374    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-data\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.034565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.034477    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-credential-socket\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.034565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.034508    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-token\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.149492    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-socket\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.149558    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-envoy\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.149583    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-data\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.149654    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-credential-socket\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.149688    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-token\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.149760    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4113087-b863-4c49-9942-bfbbc4115fb8-istiod-ca-cert\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.149785    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjl7\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-kube-api-access-2sjl7\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.149982    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-podinfo\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.150033    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-certs\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.150171    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-data\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.150344    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-certs\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150562 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.150418    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-socket\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.150776 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.150572    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-credential-socket\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.163853 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.151975    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4113087-b863-4c49-9942-bfbbc4115fb8-istiod-ca-cert\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.195942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.173551    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-token\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.196941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.196894    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-podinfo\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.209974 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.209942    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjl7\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-kube-api-access-2sjl7\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.210380 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.210357    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-envoy\") pod \"e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") " pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.288459 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.288164    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:09.664528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.664494    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"]
Nov 03 10:56:09.668245 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:09.668212    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4113087_b863_4c49_9942_bfbbc4115fb8.slice/crio-743f78df521c1ce8b2f83646f360e9ca78acb23c660f353ae481d7b90d14a9ea WatchSource:0}: Error finding container 743f78df521c1ce8b2f83646f360e9ca78acb23c660f353ae481d7b90d14a9ea: Status 404 returned error can't find the container with id 743f78df521c1ce8b2f83646f360e9ca78acb23c660f353ae481d7b90d14a9ea
Nov 03 10:56:09.671019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.670983    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:09.671117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.671057    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:09.671117 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:09.671093    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:10.273985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:10.273942    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" event={"ID":"c4113087-b863-4c49-9942-bfbbc4115fb8","Type":"ContainerStarted","Data":"43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c"}
Nov 03 10:56:10.274553 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:10.274014    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" event={"ID":"c4113087-b863-4c49-9942-bfbbc4115fb8","Type":"ContainerStarted","Data":"743f78df521c1ce8b2f83646f360e9ca78acb23c660f353ae481d7b90d14a9ea"}
Nov 03 10:56:10.289099 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:10.289068    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:10.290871 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:10.290852    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:56:10.290964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:10.290896    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:56:10.296521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:10.296454    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podStartSLOduration=2.296436969 podStartE2EDuration="2.296436969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:10.292386808 +0000 UTC m=+1614.197241253" watchObservedRunningTime="2026-11-03 10:56:10.296436969 +0000 UTC m=+1614.201291412"
Nov 03 10:56:11.293894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:11.293866    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:12.286287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:12.286257    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:12.287376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:12.287355    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:56:13.190149 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:56:13.190116    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:56:17.169804 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:17.169777    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 10:56:17.170201 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:17.169842    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 10:56:17.977397 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:17.977360    5393 scope.go:122] "RemoveContainer" containerID="abaae3c7bf1e56f8d9f0b183252c53933e9943ebc2584537e4517188b85dde03"
Nov 03 10:56:26.070085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:26.069053    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.195617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:26.167592    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 03 10:56:26.195617 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:26.167874    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.211735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:26.211674    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:26.336997 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:26.298124    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 10:56:26.749461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:26.749072    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:56:26.749461 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:26.749152    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:31.735253 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:31.735216    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"]
Nov 03 10:56:31.735901 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:31.735865    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" containerID="cri-o://bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2" gracePeriod=30
Nov 03 10:56:31.989484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:31.989090    5393 patch_prober.go:29] interesting pod/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 10:56:31.989484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:31.989164    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 10:56:32.266224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.266146    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"]
Nov 03 10:56:32.272388 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.272360    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:32.275245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.275219    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"]
Nov 03 10:56:32.357946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.353291    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-fcc2ce\""
Nov 03 10:56:32.383947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.360206    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-fcc2ce-dockercfg-bmkhh\""
Nov 03 10:56:32.410831 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.410781    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b17534-222e-4854-8bfc-7bd86b8af595-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t\" (UID: \"69b17534-222e-4854-8bfc-7bd86b8af595\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:32.411327 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.411302    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd89c\" (UniqueName: \"kubernetes.io/projected/69b17534-222e-4854-8bfc-7bd86b8af595-kube-api-access-jd89c\") pod \"payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t\" (UID: \"69b17534-222e-4854-8bfc-7bd86b8af595\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:32.516851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.516759    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd89c\" (UniqueName: \"kubernetes.io/projected/69b17534-222e-4854-8bfc-7bd86b8af595-kube-api-access-jd89c\") pod \"payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t\" (UID: \"69b17534-222e-4854-8bfc-7bd86b8af595\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:32.528243 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.528210    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b17534-222e-4854-8bfc-7bd86b8af595-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t\" (UID: \"69b17534-222e-4854-8bfc-7bd86b8af595\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:32.553409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.553071    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b17534-222e-4854-8bfc-7bd86b8af595-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t\" (UID: \"69b17534-222e-4854-8bfc-7bd86b8af595\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:32.579483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.579295    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd89c\" (UniqueName: \"kubernetes.io/projected/69b17534-222e-4854-8bfc-7bd86b8af595-kube-api-access-jd89c\") pod \"payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t\" (UID: \"69b17534-222e-4854-8bfc-7bd86b8af595\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:32.606203 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.606163    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:32.966654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:32.966625    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"]
Nov 03 10:56:32.967618 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:32.967592    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b17534_222e_4854_8bfc_7bd86b8af595.slice/crio-e9fb1812d4ca4c283136de1633ba5e63daa5cfa766ddf1df7ee32b65c2268a96 WatchSource:0}: Error finding container e9fb1812d4ca4c283136de1633ba5e63daa5cfa766ddf1df7ee32b65c2268a96: Status 404 returned error can't find the container with id e9fb1812d4ca4c283136de1633ba5e63daa5cfa766ddf1df7ee32b65c2268a96
Nov 03 10:56:33.512283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.512249    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"]
Nov 03 10:56:33.517438 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.517408    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.524814 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.524545    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-cc8214-openshift-default-dockercfg-z4fsr\""
Nov 03 10:56:33.527402 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.527376    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"]
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693019    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-token\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693107    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693236    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-credential-socket\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693272    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-envoy\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693314    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-data\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693407    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-socket\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693508    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-certs\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693542    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-podinfo\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.693944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.693627    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpj9j\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-kube-api-access-vpj9j\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.806771 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.800471    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-socket\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.817820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.817778    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-certs\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.823598 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.823571    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-podinfo\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.824304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.818659    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-certs\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.824495 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.820635    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t" event={"ID":"69b17534-222e-4854-8bfc-7bd86b8af595","Type":"ContainerStarted","Data":"59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9"}
Nov 03 10:56:33.824807 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.806679    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-socket\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.825196 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.825173    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpj9j\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-kube-api-access-vpj9j\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.825667 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.825651    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:33.826034 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.826003    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-token\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.826214 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.826195    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.826870 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.826845    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t" event={"ID":"69b17534-222e-4854-8bfc-7bd86b8af595","Type":"ContainerStarted","Data":"e9fb1812d4ca4c283136de1633ba5e63daa5cfa766ddf1df7ee32b65c2268a96"}
Nov 03 10:56:33.828514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.828345    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-credential-socket\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.829041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.829021    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-credential-socket\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.829217 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.829200    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-envoy\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.829389 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.829372    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-data\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.830251 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.830225    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-data\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.830346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.829905    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istiod-ca-cert\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.853942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.850859    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-podinfo\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.859939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.856596    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t" podStartSLOduration=1.856574689 podStartE2EDuration="1.856574689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:33.851725141 +0000 UTC m=+1637.756579583" watchObservedRunningTime="2026-11-03 10:56:33.856574689 +0000 UTC m=+1637.761429131"
Nov 03 10:56:33.905412 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.900500    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-envoy\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.905412 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.901015    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-token\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:33.905412 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:33.901595    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpj9j\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-kube-api-access-vpj9j\") pod \"e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") " pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:34.146793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:34.146761    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:34.443100 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:34.443072    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"]
Nov 03 10:56:34.443297 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:34.443268    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7724d8_aa88_49c8_9a08_844b43f5a00d.slice/crio-82bbd8940470540c66017bcbe5030b5f7bd50cab16a524f70a61ed8fa8aa0544 WatchSource:0}: Error finding container 82bbd8940470540c66017bcbe5030b5f7bd50cab16a524f70a61ed8fa8aa0544: Status 404 returned error can't find the container with id 82bbd8940470540c66017bcbe5030b5f7bd50cab16a524f70a61ed8fa8aa0544
Nov 03 10:56:34.446407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:34.446363    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:34.446517 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:34.446435    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:34.446517 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:34.446466    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:34.832650 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:34.832543    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" event={"ID":"4c7724d8-aa88-49c8-9a08-844b43f5a00d","Type":"ContainerStarted","Data":"c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4"}
Nov 03 10:56:34.832650 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:34.832614    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" event={"ID":"4c7724d8-aa88-49c8-9a08-844b43f5a00d","Type":"ContainerStarted","Data":"82bbd8940470540c66017bcbe5030b5f7bd50cab16a524f70a61ed8fa8aa0544"}
Nov 03 10:56:34.857102 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:34.857045    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" podStartSLOduration=1.857026506 podStartE2EDuration="1.857026506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:34.852134122 +0000 UTC m=+1638.756988562" watchObservedRunningTime="2026-11-03 10:56:34.857026506 +0000 UTC m=+1638.761880949"
Nov 03 10:56:35.147276 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:35.147239    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:35.155055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:35.155025    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:35.839271 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:35.839242    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:35.840165 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:35.840145    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:56:37.106691 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.106663    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:56:37.123454 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123415    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-socket\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.123658 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123468    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-certs\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.123658 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123568    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-envoy\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.123796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123657    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-podinfo\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.123796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123695    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istiod-ca-cert\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.123796 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123757    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-socket" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:37.124003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123767    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-credential-socket\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.124003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123825    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-certs" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:37.124003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123946    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-token\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-token\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.124003 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.123988    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-credential-socket" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:37.124224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.124031    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-kube-api-access-cchn9\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-kube-api-access-cchn9\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.124224 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.124081    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-data\") pod \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\" (UID: \"5d4f9c29-b268-4c8f-be71-f398b2978ba4\") "
Nov 03 10:56:37.124357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.124288    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istiod-ca-cert" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:37.124521 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.124501    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-data" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:37.125129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.125059    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.125129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.125084    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.125129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.125097    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.125129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.125112    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.125129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.125126    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.126713 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.126682    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-kube-api-access-cchn9" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "kube-api-access-cchn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:37.127097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.127071    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-podinfo" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:56:37.127315 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.127276    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-token" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:37.132071 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.127413    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-envoy" pod "5d4f9c29-b268-4c8f-be71-f398b2978ba4" (UID: "5d4f9c29-b268-4c8f-be71-f398b2978ba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:56:37.225988 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.225910    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cchn9\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-kube-api-access-cchn9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.226268 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.226240    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.226268 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.226269    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.226450 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.226282    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d4f9c29-b268-4c8f-be71-f398b2978ba4-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:37.853409 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.853363    5393 generic.go:356] "Generic (PLEG): container finished" podID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerID="bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2" exitCode=0
Nov 03 10:56:37.853607 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.853453    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" event={"ID":"5d4f9c29-b268-4c8f-be71-f398b2978ba4","Type":"ContainerDied","Data":"bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2"}
Nov 03 10:56:37.853607 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.853555    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd" event={"ID":"5d4f9c29-b268-4c8f-be71-f398b2978ba4","Type":"ContainerDied","Data":"7a80dd80eabb78093aef7972270598a9311c08f0e4997deb2a16753ff427d8b3"}
Nov 03 10:56:37.853607 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.853466    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"
Nov 03 10:56:37.853607 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.853589    5393 scope.go:122] "RemoveContainer" containerID="bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2"
Nov 03 10:56:37.913985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.913887    5393 scope.go:122] "RemoveContainer" containerID="48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f"
Nov 03 10:56:37.950959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.947514    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"]
Nov 03 10:56:37.954350 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.954044    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-cd6d01-openshift-default-85bbcffdd5-pglvd"]
Nov 03 10:56:37.963305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.963287    5393 scope.go:122] "RemoveContainer" containerID="bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2"
Nov 03 10:56:37.963703 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:56:37.963683    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2\": container with ID starting with bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2 not found: ID does not exist" containerID="bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2"
Nov 03 10:56:37.963759 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.963717    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2"} err="failed to get container status \"bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2\": rpc error: code = NotFound desc = could not find container \"bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2\": container with ID starting with bc98e60f015e77a478ae7e9bb30049b52c55006e0323f5e309a8b6beaa774ee2 not found: ID does not exist"
Nov 03 10:56:37.963759 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.963739    5393 scope.go:122] "RemoveContainer" containerID="48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f"
Nov 03 10:56:37.964049 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:56:37.964029    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f\": container with ID starting with 48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f not found: ID does not exist" containerID="48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f"
Nov 03 10:56:37.964107 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:37.964061    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f"} err="failed to get container status \"48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f\": rpc error: code = NotFound desc = could not find container \"48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f\": container with ID starting with 48a3fc8415f690ed043b7d7fc493f36995e0c9fea318523e960488b06235371f not found: ID does not exist"
Nov 03 10:56:38.232197 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.232126    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" path="/var/lib/kubelet/pods/5d4f9c29-b268-4c8f-be71-f398b2978ba4/volumes"
Nov 03 10:56:38.389344 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.389308    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"]
Nov 03 10:56:38.390317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.390292    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy"
Nov 03 10:56:38.390317 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.390317    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy"
Nov 03 10:56:38.390532 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.390512    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy"
Nov 03 10:56:38.390532 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.390527    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy"
Nov 03 10:56:38.395833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.395816    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:38.402097 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.402074    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-cc8214-dockercfg-hsx2p\""
Nov 03 10:56:38.405111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.405087    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"]
Nov 03 10:56:38.478941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.478227    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcr7\" (UniqueName: \"kubernetes.io/projected/941c0d96-a831-41d5-9392-7802117b42ad-kube-api-access-tvcr7\") pod \"payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m\" (UID: \"941c0d96-a831-41d5-9392-7802117b42ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:38.478941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.478342    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941c0d96-a831-41d5-9392-7802117b42ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m\" (UID: \"941c0d96-a831-41d5-9392-7802117b42ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:38.478941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.478701    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-cc8214\""
Nov 03 10:56:38.604000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.601901    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941c0d96-a831-41d5-9392-7802117b42ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m\" (UID: \"941c0d96-a831-41d5-9392-7802117b42ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:38.604000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.602225    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcr7\" (UniqueName: \"kubernetes.io/projected/941c0d96-a831-41d5-9392-7802117b42ad-kube-api-access-tvcr7\") pod \"payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m\" (UID: \"941c0d96-a831-41d5-9392-7802117b42ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:38.604000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.603628    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941c0d96-a831-41d5-9392-7802117b42ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m\" (UID: \"941c0d96-a831-41d5-9392-7802117b42ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:38.641942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.641540    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcr7\" (UniqueName: \"kubernetes.io/projected/941c0d96-a831-41d5-9392-7802117b42ad-kube-api-access-tvcr7\") pod \"payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m\" (UID: \"941c0d96-a831-41d5-9392-7802117b42ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:38.712329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:38.712294    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:39.011182 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:39.011154    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"]
Nov 03 10:56:39.011933 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:39.011894    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941c0d96_a831_41d5_9392_7802117b42ad.slice/crio-543062c540c944ee7a36bebd3505fe6575f4fdeab80424252247f65f7ef813a9 WatchSource:0}: Error finding container 543062c540c944ee7a36bebd3505fe6575f4fdeab80424252247f65f7ef813a9: Status 404 returned error can't find the container with id 543062c540c944ee7a36bebd3505fe6575f4fdeab80424252247f65f7ef813a9
Nov 03 10:56:39.976573 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:39.975108    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m" event={"ID":"941c0d96-a831-41d5-9392-7802117b42ad","Type":"ContainerStarted","Data":"bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1"}
Nov 03 10:56:39.976573 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:39.975188    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m" event={"ID":"941c0d96-a831-41d5-9392-7802117b42ad","Type":"ContainerStarted","Data":"543062c540c944ee7a36bebd3505fe6575f4fdeab80424252247f65f7ef813a9"}
Nov 03 10:56:39.979731 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:39.978693    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:39.997334 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:39.997285    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m" podStartSLOduration=1.997268133 podStartE2EDuration="1.997268133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:39.993313884 +0000 UTC m=+1643.898168338" watchObservedRunningTime="2026-11-03 10:56:39.997268133 +0000 UTC m=+1643.902122611"
Nov 03 10:56:42.068945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:42.068030    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded"
Nov 03 10:56:42.605945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:42.605611    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 10:56:42.644944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:42.644543    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:42.870098 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:42.869975    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"]
Nov 03 10:56:42.878947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:42.873095    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"]
Nov 03 10:56:43.163998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:43.126044    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.171998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:43.168199    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 10:56:43.262230 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:43.262185    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.608396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:43.607030    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.627957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:43.608853    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:56:43.656103 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:43.656036    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:56:43.670041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:43.661749    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:56:43.671411 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:56:43.671378    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:56:44.004998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:44.003322    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 10:56:44.075959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:44.075890    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 10:56:44.836358 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:44.836331    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 10:56:45.136811 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.136775    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"]
Nov 03 10:56:45.137380 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.137365    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy"
Nov 03 10:56:45.137430 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.137381    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4f9c29-b268-4c8f-be71-f398b2978ba4" containerName="istio-proxy"
Nov 03 10:56:45.141150 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.141135    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:45.148137 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.148114    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-47a8a4\""
Nov 03 10:56:45.150364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.150343    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-47a8a4-dockercfg-dnf2l\""
Nov 03 10:56:45.152335 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.152312    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"]
Nov 03 10:56:45.169739 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.169708    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"]
Nov 03 10:56:45.259946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.231305    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"]
Nov 03 10:56:45.259946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.231499    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:45.383962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.376717    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519bd00-775f-484a-bb47-091b12fee81c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld\" (UID: \"8519bd00-775f-484a-bb47-091b12fee81c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:45.391991 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.384513    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dkl\" (UniqueName: \"kubernetes.io/projected/8519bd00-775f-484a-bb47-091b12fee81c-kube-api-access-97dkl\") pod \"payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld\" (UID: \"8519bd00-775f-484a-bb47-091b12fee81c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:45.495376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.494321    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519bd00-775f-484a-bb47-091b12fee81c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld\" (UID: \"8519bd00-775f-484a-bb47-091b12fee81c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:45.495376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.494574    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dkl\" (UniqueName: \"kubernetes.io/projected/8519bd00-775f-484a-bb47-091b12fee81c-kube-api-access-97dkl\") pod \"payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld\" (UID: \"8519bd00-775f-484a-bb47-091b12fee81c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:45.495376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.494647    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bb4\" (UniqueName: \"kubernetes.io/projected/0c3ad30c-e2ac-4f38-a031-71f228f0f522-kube-api-access-k7bb4\") pod \"payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr\" (UID: \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:45.495376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.494749    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3ad30c-e2ac-4f38-a031-71f228f0f522-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr\" (UID: \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:45.495376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.495330    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519bd00-775f-484a-bb47-091b12fee81c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld\" (UID: \"8519bd00-775f-484a-bb47-091b12fee81c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:45.510941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.508005    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dkl\" (UniqueName: \"kubernetes.io/projected/8519bd00-775f-484a-bb47-091b12fee81c-kube-api-access-97dkl\") pod \"payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld\" (UID: \"8519bd00-775f-484a-bb47-091b12fee81c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:45.602947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.599777    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bb4\" (UniqueName: \"kubernetes.io/projected/0c3ad30c-e2ac-4f38-a031-71f228f0f522-kube-api-access-k7bb4\") pod \"payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr\" (UID: \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:45.602947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.599901    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3ad30c-e2ac-4f38-a031-71f228f0f522-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr\" (UID: \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:45.602947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.601054    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3ad30c-e2ac-4f38-a031-71f228f0f522-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr\" (UID: \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:45.615329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.615297    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bb4\" (UniqueName: \"kubernetes.io/projected/0c3ad30c-e2ac-4f38-a031-71f228f0f522-kube-api-access-k7bb4\") pod \"payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr\" (UID: \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:45.757881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.757793    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:45.843985 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:45.840221    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:46.131846 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.131812    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"]
Nov 03 10:56:46.336022 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.335988    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"]
Nov 03 10:56:46.337214 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:46.337190    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3ad30c_e2ac_4f38_a031_71f228f0f522.slice/crio-4e9f42e8417bc4b3e17a0dbb21eff3e27e64b7c56eaa4de67cd70af4eaad92f1 WatchSource:0}: Error finding container 4e9f42e8417bc4b3e17a0dbb21eff3e27e64b7c56eaa4de67cd70af4eaad92f1: Status 404 returned error can't find the container with id 4e9f42e8417bc4b3e17a0dbb21eff3e27e64b7c56eaa4de67cd70af4eaad92f1
Nov 03 10:56:46.532511 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.532400    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld" event={"ID":"8519bd00-775f-484a-bb47-091b12fee81c","Type":"ContainerStarted","Data":"a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb"}
Nov 03 10:56:46.532511 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.532458    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld" event={"ID":"8519bd00-775f-484a-bb47-091b12fee81c","Type":"ContainerStarted","Data":"0fd93efd14d272a5bd32fc61b55c1a1a46df72076434e54977ebcc55152235dd"}
Nov 03 10:56:46.532511 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.532483    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:46.534089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.534064    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr" event={"ID":"0c3ad30c-e2ac-4f38-a031-71f228f0f522","Type":"ContainerStarted","Data":"018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761"}
Nov 03 10:56:46.534222 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.534107    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr" event={"ID":"0c3ad30c-e2ac-4f38-a031-71f228f0f522","Type":"ContainerStarted","Data":"4e9f42e8417bc4b3e17a0dbb21eff3e27e64b7c56eaa4de67cd70af4eaad92f1"}
Nov 03 10:56:46.534222 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.534166    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:46.550223 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.550181    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld" podStartSLOduration=1.55016863 podStartE2EDuration="1.55016863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:46.547781351 +0000 UTC m=+1650.452635793" watchObservedRunningTime="2026-11-03 10:56:46.55016863 +0000 UTC m=+1650.455023071"
Nov 03 10:56:46.566634 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:46.566586    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr" podStartSLOduration=1.5665724349999999 podStartE2EDuration="1.566572435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:46.565239863 +0000 UTC m=+1650.470094305" watchObservedRunningTime="2026-11-03 10:56:46.566572435 +0000 UTC m=+1650.471426867"
Nov 03 10:56:47.871496 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:47.871437    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx" podUID="92f6573b-c38d-430f-9634-6fd8823cad2e" containerName="payload-processing" containerID="cri-o://d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00" gracePeriod=25
Nov 03 10:56:47.874015 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:47.873982    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8" podUID="b405dc6a-ff68-46db-a6a5-21b93e6c094d" containerName="payload-pre-processing" containerID="cri-o://fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8" gracePeriod=25
Nov 03 10:56:48.266059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.265792    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:56:48.289681 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.289656    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:56:48.344306 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.344275    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f6573b-c38d-430f-9634-6fd8823cad2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f6573b-c38d-430f-9634-6fd8823cad2e-plugins-config-volume\") pod \"92f6573b-c38d-430f-9634-6fd8823cad2e\" (UID: \"92f6573b-c38d-430f-9634-6fd8823cad2e\") "
Nov 03 10:56:48.344479 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.344373    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b405dc6a-ff68-46db-a6a5-21b93e6c094d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b405dc6a-ff68-46db-a6a5-21b93e6c094d-plugins-config-volume\") pod \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\" (UID: \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\") "
Nov 03 10:56:48.344564 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.344541    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f6573b-c38d-430f-9634-6fd8823cad2e-kube-api-access-m4vhc\" (UniqueName: \"kubernetes.io/projected/92f6573b-c38d-430f-9634-6fd8823cad2e-kube-api-access-m4vhc\") pod \"92f6573b-c38d-430f-9634-6fd8823cad2e\" (UID: \"92f6573b-c38d-430f-9634-6fd8823cad2e\") "
Nov 03 10:56:48.344628 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.344584    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b405dc6a-ff68-46db-a6a5-21b93e6c094d-kube-api-access-vgrvn\" (UniqueName: \"kubernetes.io/projected/b405dc6a-ff68-46db-a6a5-21b93e6c094d-kube-api-access-vgrvn\") pod \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\" (UID: \"b405dc6a-ff68-46db-a6a5-21b93e6c094d\") "
Nov 03 10:56:48.344802 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.344777    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b405dc6a-ff68-46db-a6a5-21b93e6c094d-plugins-config-volume" pod "b405dc6a-ff68-46db-a6a5-21b93e6c094d" (UID: "b405dc6a-ff68-46db-a6a5-21b93e6c094d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:48.345088 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.345058    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f6573b-c38d-430f-9634-6fd8823cad2e-plugins-config-volume" pod "92f6573b-c38d-430f-9634-6fd8823cad2e" (UID: "92f6573b-c38d-430f-9634-6fd8823cad2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:56:48.345456 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.345427    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f6573b-c38d-430f-9634-6fd8823cad2e-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:48.345555 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.345461    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b405dc6a-ff68-46db-a6a5-21b93e6c094d-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:48.347455 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.347432    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b405dc6a-ff68-46db-a6a5-21b93e6c094d-kube-api-access-vgrvn" pod "b405dc6a-ff68-46db-a6a5-21b93e6c094d" (UID: "b405dc6a-ff68-46db-a6a5-21b93e6c094d"). InnerVolumeSpecName "kube-api-access-vgrvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:48.347753 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.347729    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f6573b-c38d-430f-9634-6fd8823cad2e-kube-api-access-m4vhc" pod "92f6573b-c38d-430f-9634-6fd8823cad2e" (UID: "92f6573b-c38d-430f-9634-6fd8823cad2e"). InnerVolumeSpecName "kube-api-access-m4vhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:56:48.389669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.389438    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"]
Nov 03 10:56:48.390273 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.390248    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b405dc6a-ff68-46db-a6a5-21b93e6c094d" containerName="payload-pre-processing"
Nov 03 10:56:48.390273 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.390272    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b405dc6a-ff68-46db-a6a5-21b93e6c094d" containerName="payload-pre-processing"
Nov 03 10:56:48.390460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.390300    5393 cpu_manager.go:414] "containerMap: removing container" podUID="92f6573b-c38d-430f-9634-6fd8823cad2e" containerName="payload-processing"
Nov 03 10:56:48.390460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.390307    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f6573b-c38d-430f-9634-6fd8823cad2e" containerName="payload-processing"
Nov 03 10:56:48.390749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.390623    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b405dc6a-ff68-46db-a6a5-21b93e6c094d" containerName="payload-pre-processing"
Nov 03 10:56:48.390749 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.390648    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f6573b-c38d-430f-9634-6fd8823cad2e" containerName="payload-processing"
Nov 03 10:56:48.394608 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.394579    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.404283 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.404201    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"]
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.446466    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8g5\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-kube-api-access-vd8g5\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.446642    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.446684    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a50084-678c-42e3-b5be-bbc87cfd0574-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.446774    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.446840    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.446866    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.446931    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.446981    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.447021    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.447110    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vhc\" (UniqueName: \"kubernetes.io/projected/92f6573b-c38d-430f-9634-6fd8823cad2e-kube-api-access-m4vhc\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:48.447287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.447127    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgrvn\" (UniqueName: \"kubernetes.io/projected/b405dc6a-ff68-46db-a6a5-21b93e6c094d-kube-api-access-vgrvn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:56:48.449085 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.449054    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"]
Nov 03 10:56:48.449212 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.449111    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"]
Nov 03 10:56:48.449285 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.449214    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.534940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.533007    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"]
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.549342    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.549434    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a50084-678c-42e3-b5be-bbc87cfd0574-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.549595    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.549671    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.549739    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.549817    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.549854    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvscj\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-kube-api-access-vvscj\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.549885    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550000    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550041    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550080    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550116    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550156    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550217    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.550348 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550355    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.552056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550396    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.552056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550443    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.552056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.550479    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8g5\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-kube-api-access-vd8g5\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.552467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.552435    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.552742 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.552719    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.560272 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.560023    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.562636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.562076    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a50084-678c-42e3-b5be-bbc87cfd0574-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.562636 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.562405    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.575700 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.572197    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.575700 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.574597    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.582388 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.582354    5393 generic.go:356] "Generic (PLEG): container finished" podID="92f6573b-c38d-430f-9634-6fd8823cad2e" containerID="d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00" exitCode=0
Nov 03 10:56:48.584192 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.582447    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"
Nov 03 10:56:48.584353 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.582467    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx" event={"ID":"92f6573b-c38d-430f-9634-6fd8823cad2e","Type":"ContainerDied","Data":"d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00"}
Nov 03 10:56:48.584454 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.584436    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"]
Nov 03 10:56:48.584525 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.584485    5393 scope.go:122] "RemoveContainer" containerID="d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00"
Nov 03 10:56:48.584851 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.584830    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx" event={"ID":"92f6573b-c38d-430f-9634-6fd8823cad2e","Type":"ContainerDied","Data":"00e14136a8de21c46d4b00ffac59b91bb3bbb6ee42ca7eb78c929f5f5270ea3d"}
Nov 03 10:56:48.586881 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.586857    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8g5\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-kube-api-access-vd8g5\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.587775 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.587755    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ce5555-openshift-default-dockercfg-2h6ft\""
Nov 03 10:56:48.595839 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.595570    5393 generic.go:356] "Generic (PLEG): container finished" podID="b405dc6a-ff68-46db-a6a5-21b93e6c094d" containerID="fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8" exitCode=0
Nov 03 10:56:48.595839 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.595717    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"
Nov 03 10:56:48.595839 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.595729    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8" event={"ID":"b405dc6a-ff68-46db-a6a5-21b93e6c094d","Type":"ContainerDied","Data":"fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8"}
Nov 03 10:56:48.595839 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.595794    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8" event={"ID":"b405dc6a-ff68-46db-a6a5-21b93e6c094d","Type":"ContainerDied","Data":"62ee253423aafe6bdf552ef496b35efe22679d316c528f6776f3fcc895db0f9f"}
Nov 03 10:56:48.604903 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.604870    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.607384 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.607355    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.644995 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.644676    5393 scope.go:122] "RemoveContainer" containerID="d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00"
Nov 03 10:56:48.645653 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:56:48.645619    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00\": container with ID starting with d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00 not found: ID does not exist" containerID="d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00"
Nov 03 10:56:48.645765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.645655    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00"} err="failed to get container status \"d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00\": rpc error: code = NotFound desc = could not find container \"d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00\": container with ID starting with d570739a1632be7b46fd5d0a1d86c2109dc8ec372c537a66df7b8686b7a81c00 not found: ID does not exist"
Nov 03 10:56:48.645765 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.645673    5393 scope.go:122] "RemoveContainer" containerID="fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8"
Nov 03 10:56:48.651491 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651464    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-data\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.651613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651507    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istiod-ca-cert\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.651613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651534    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556h6\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-kube-api-access-556h6\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.651613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651555    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-envoy\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.651613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651611    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.651833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651626    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.651833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651641    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-token\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.651833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651660    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.651833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651695    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.651833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651714    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvscj\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-kube-api-access-vvscj\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.651833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651762    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.651833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651784    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.651833 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651819    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-credential-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.652270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651844    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.652270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651860    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.652270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651887    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.652270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651947    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-podinfo\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.652270 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.651967    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-certs\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.652526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.652315    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-data\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.653073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.652754    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-certs\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.653073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.652955    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-credential-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.653073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.653063    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-socket\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.656011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.653551    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istiod-ca-cert\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.656011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.655490    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-podinfo\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.656011 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.655865    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-envoy\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.660585 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.660476    5393 scope.go:122] "RemoveContainer" containerID="fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8"
Nov 03 10:56:48.660798 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:56:48.660776    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8\": container with ID starting with fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8 not found: ID does not exist" containerID="fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8"
Nov 03 10:56:48.660870 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.660803    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8"} err="failed to get container status \"fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8\": rpc error: code = NotFound desc = could not find container \"fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8\": container with ID starting with fddbb49eba3c15c4bdab978db96e1d246c5845040120e51351e2df8480dd4ea8 not found: ID does not exist"
Nov 03 10:56:48.668792 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.668741    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"]
Nov 03 10:56:48.671119 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.671063    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvscj\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-kube-api-access-vvscj\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.682470 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.682444    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a35a97-6bff94d6bc-smzl8"]
Nov 03 10:56:48.684396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.684370    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-token\") pod \"e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") " pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.708664 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.708632    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"]
Nov 03 10:56:48.718968 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.718932    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a35a97-5bd6769d89-k5lzx"]
Nov 03 10:56:48.754056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.754014    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-data\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.754359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.754331    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istiod-ca-cert\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.754548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.754528    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556h6\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-kube-api-access-556h6\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.754723 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.754700    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-envoy\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.768089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.768051    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-token\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.768581 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.768557    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-credential-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.768753 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.768737    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.790944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.768946    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-podinfo\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.791268 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.791236    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-certs\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.791536 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.766630    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-data\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.791644 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.785086    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-credential-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.791743 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.767314    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istiod-ca-cert\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.791840 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.785302    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-socket\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.791951 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.766181    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:48.792874 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.792545    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-certs\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.792874 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.792803    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:48.949809 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.934080    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-podinfo\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.951599 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.951539    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-envoy\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.955798 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.955722    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-token\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:48.963640 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:48.963574    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-556h6\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-kube-api-access-556h6\") pod \"e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") " pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:49.206667 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.206583    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:49.400958 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:49.398730    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a50084_678c_42e3_b5be_bbc87cfd0574.slice/crio-b4f52989e115d412c2ce01717b3435e84c8be23bff9b1e01e9ea212e0b5022ad WatchSource:0}: Error finding container b4f52989e115d412c2ce01717b3435e84c8be23bff9b1e01e9ea212e0b5022ad: Status 404 returned error can't find the container with id b4f52989e115d412c2ce01717b3435e84c8be23bff9b1e01e9ea212e0b5022ad
Nov 03 10:56:49.403787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.402351    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.403787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.402449    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.403787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.402490    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.407308 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.405504    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"]
Nov 03 10:56:49.410391 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:49.410360    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b47fb10_dfcd_4463_bc70_889f926ce1d2.slice/crio-cb837bdecdaaf2b35e15b9adc6f70571f348b9f264ad089e5d32bccc75f84fbf WatchSource:0}: Error finding container cb837bdecdaaf2b35e15b9adc6f70571f348b9f264ad089e5d32bccc75f84fbf: Status 404 returned error can't find the container with id cb837bdecdaaf2b35e15b9adc6f70571f348b9f264ad089e5d32bccc75f84fbf
Nov 03 10:56:49.411000 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.410974    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"]
Nov 03 10:56:49.419396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.419351    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.419490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.419439    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.419528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.419489    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.505398 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:56:49.505363    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4bbcff_54aa_47e7_ae55_1e451c34a943.slice/crio-4f37d6c023e2d02e9e30986b58ca7e7839c984041a61a20a66bcb345262de188 WatchSource:0}: Error finding container 4f37d6c023e2d02e9e30986b58ca7e7839c984041a61a20a66bcb345262de188: Status 404 returned error can't find the container with id 4f37d6c023e2d02e9e30986b58ca7e7839c984041a61a20a66bcb345262de188
Nov 03 10:56:49.505886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.505862    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"]
Nov 03 10:56:49.508287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.508247    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.508395 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.508330    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.508395 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.508373    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:49.645956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.643048    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" event={"ID":"1b47fb10-dfcd-4463-bc70-889f926ce1d2","Type":"ContainerStarted","Data":"4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb"}
Nov 03 10:56:49.645956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.643126    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" event={"ID":"1b47fb10-dfcd-4463-bc70-889f926ce1d2","Type":"ContainerStarted","Data":"cb837bdecdaaf2b35e15b9adc6f70571f348b9f264ad089e5d32bccc75f84fbf"}
Nov 03 10:56:49.667955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.664899    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" event={"ID":"00a50084-678c-42e3-b5be-bbc87cfd0574","Type":"ContainerStarted","Data":"9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad"}
Nov 03 10:56:49.677073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.670660    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" event={"ID":"00a50084-678c-42e3-b5be-bbc87cfd0574","Type":"ContainerStarted","Data":"b4f52989e115d412c2ce01717b3435e84c8be23bff9b1e01e9ea212e0b5022ad"}
Nov 03 10:56:49.677073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.674174    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podStartSLOduration=1.6741548659999999 podStartE2EDuration="1.674154866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:49.672364399 +0000 UTC m=+1653.577218842" watchObservedRunningTime="2026-11-03 10:56:49.674154866 +0000 UTC m=+1653.579009309"
Nov 03 10:56:49.679062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.679022    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" event={"ID":"2e4bbcff-54aa-47e7-ae55-1e451c34a943","Type":"ContainerStarted","Data":"4f37d6c023e2d02e9e30986b58ca7e7839c984041a61a20a66bcb345262de188"}
Nov 03 10:56:49.703425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.703363    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podStartSLOduration=1.703346366 podStartE2EDuration="1.703346366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:49.702284661 +0000 UTC m=+1653.607139103" watchObservedRunningTime="2026-11-03 10:56:49.703346366 +0000 UTC m=+1653.608200808"
Nov 03 10:56:49.793125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.792737    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:49.793125 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:49.793021    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:50.263136 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:50.261292    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f6573b-c38d-430f-9634-6fd8823cad2e" path="/var/lib/kubelet/pods/92f6573b-c38d-430f-9634-6fd8823cad2e/volumes"
Nov 03 10:56:50.263634 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:50.263229    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b405dc6a-ff68-46db-a6a5-21b93e6c094d" path="/var/lib/kubelet/pods/b405dc6a-ff68-46db-a6a5-21b93e6c094d/volumes"
Nov 03 10:56:50.753202 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:50.753169    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" event={"ID":"2e4bbcff-54aa-47e7-ae55-1e451c34a943","Type":"ContainerStarted","Data":"781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da"}
Nov 03 10:56:50.773465 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:50.773394    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" podStartSLOduration=2.773380576 podStartE2EDuration="2.773380576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:56:50.771955286 +0000 UTC m=+1654.676809725" watchObservedRunningTime="2026-11-03 10:56:50.773380576 +0000 UTC m=+1654.678235017"
Nov 03 10:56:50.812944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:50.811760    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 10:56:50.812944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:50.811819    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 10:56:50.828942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:50.828304    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 10:56:50.828942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:50.828378    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 10:56:51.011241 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.011121    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 10:56:51.215957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.215848    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:51.244953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.244618    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:51.779538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.769979    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:51.803973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.796498    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 10:56:51.803973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.796557    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 10:56:51.803973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.797003    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 10:56:51.803973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.797037    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 10:56:51.803973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:51.801177    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:56:52.799946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:52.794559    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 10:56:52.799946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:52.794620    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 10:56:52.799946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:52.795078    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 10:56:52.799946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:52.795116    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 10:56:53.801944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:53.794416    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 10:56:53.801944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:53.794476    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 10:56:53.812957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:53.812048    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 10:56:53.812957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:53.812112    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 10:56:54.804941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:54.804639    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 10:56:54.804941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:54.804697    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 10:56:54.805502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:54.805156    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 10:56:54.805502 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:54.805191    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 10:56:55.812944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:55.806416    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 10:56:55.812944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:55.806477    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 10:56:55.812944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:55.806883    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 10:56:55.812944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:55.806960    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 10:56:56.793589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:56.793388    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 10:56:56.793857 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:56.793828    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 10:56:56.794038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:56.793436    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 10:56:56.794162 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:56.794144    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 10:56:57.588543 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:57.581243    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 10:56:57.622787 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:57.622748    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 10:56:57.803952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:57.803486    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 10:56:57.803952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:57.803556    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 10:56:57.803952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:57.803954    5393 patch_prober.go:29] interesting pod/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 10:56:57.804263 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:57.803986    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 10:56:58.216318 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.215291    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": read tcp 10.134.0.2:55670->10.134.0.151:15021: read: connection reset by peer" start-of-body=
Nov 03 10:56:58.216318 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.215359    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": read tcp 10.134.0.2:55670->10.134.0.151:15021: read: connection reset by peer"
Nov 03 10:56:58.797630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.797600    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:58.798134 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.797689    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:58.798248 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.798226    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:58.798329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.798280    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:58.798949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.798893    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 10:56:58.799078 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.799017    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 10:56:58.956602 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.956569    5393 generic.go:356] "Generic (PLEG): container finished" podID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerID="43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c" exitCode=0
Nov 03 10:56:58.957057 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.957016    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" event={"ID":"c4113087-b863-4c49-9942-bfbbc4115fb8","Type":"ContainerDied","Data":"43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c"}
Nov 03 10:56:58.957589 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.957571    5393 scope.go:122] "RemoveContainer" containerID="43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c"
Nov 03 10:56:58.959527 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.959495    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:58.959627 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.959582    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:56:58.959693 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:56:58.959635    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:00.010109 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:00.009086    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" event={"ID":"c4113087-b863-4c49-9942-bfbbc4115fb8","Type":"ContainerStarted","Data":"3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82"}
Nov 03 10:57:00.289684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:00.289540    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:57:00.290073 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:00.290045    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:57:00.290237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:00.290215    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:57:01.298665 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:01.298593    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:57:01.299864 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:01.299400    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:57:01.301082 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:01.301059    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"]
Nov 03 10:57:01.301895 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:01.301505    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy" containerID="cri-o://1f24b4dd83faac2aafcbd40eacfcb654ff9b23ef8a8216329a4c21c31f3ab741" gracePeriod=30
Nov 03 10:57:02.302946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:02.298294    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:57:02.302946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:02.298355    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:57:03.291955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:03.291382    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:57:03.291955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:03.291466    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:57:04.289321 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:04.289264    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:57:04.301166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:04.301006    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:57:05.302945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:05.301308    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:57:05.302945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:05.301371    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:57:06.301970 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:06.301873    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:57:06.302291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:06.302258    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:57:06.840954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:06.836301    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:57:07.352873 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.334870    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 10:57:07.352873 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.339297    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 10:57:07.545538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.545503    5393 generic.go:356] "Generic (PLEG): container finished" podID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerID="1f24b4dd83faac2aafcbd40eacfcb654ff9b23ef8a8216329a4c21c31f3ab741" exitCode=0
Nov 03 10:57:07.545845 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.545819    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" event={"ID":"5f8cc142-76a0-4c09-85bb-e5230e83bdbc","Type":"ContainerDied","Data":"1f24b4dd83faac2aafcbd40eacfcb654ff9b23ef8a8216329a4c21c31f3ab741"}
Nov 03 10:57:07.546047 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.546031    5393 scope.go:122] "RemoveContainer" containerID="ecc7f2dff91d965b837ae2c8402c05a355bbcd64eeb9722e8922f8eec2876de0"
Nov 03 10:57:07.636420 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.636393    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.719863    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-data\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.720582    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-data" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.721031    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-podinfo\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.721103    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-socket\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.721136    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-token\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-token\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.721202    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-kube-api-access-xchq4\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-kube-api-access-xchq4\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.721241    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-envoy\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.721331    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istiod-ca-cert\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.721386    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-credential-socket\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.721533    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-certs\") pod \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\" (UID: \"5f8cc142-76a0-4c09-85bb-e5230e83bdbc\") "
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.722144    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.722352    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-certs" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.725298    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-envoy" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.725620    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istiod-ca-cert" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.725883    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-credential-socket" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:07.726695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.726290    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-socket" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:57:07.729260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.729232    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-podinfo" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:57:07.731168 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.731143    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-token" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:57:07.731816 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.731791    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-kube-api-access-xchq4" pod "5f8cc142-76a0-4c09-85bb-e5230e83bdbc" (UID: "5f8cc142-76a0-4c09-85bb-e5230e83bdbc"). InnerVolumeSpecName "kube-api-access-xchq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:57:07.823323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.823231    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:07.823323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.823267    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:07.823323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.823285    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:07.823323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.823300    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:07.823323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.823315    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:07.823323 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.823328    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:07.823704 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.823343    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xchq4\" (UniqueName: \"kubernetes.io/projected/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-kube-api-access-xchq4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:07.823704 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:07.823358    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f8cc142-76a0-4c09-85bb-e5230e83bdbc-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:57:08.311551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:08.311519    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:57:08.312312 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:08.312289    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:57:08.313038 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:08.313018    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 10:57:08.553831 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:08.553747    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f" event={"ID":"5f8cc142-76a0-4c09-85bb-e5230e83bdbc","Type":"ContainerDied","Data":"77b7bfb1554440495c1544f8fa116924adedfd4e3c281a94611f6f216c8a7925"}
Nov 03 10:57:08.553831 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:08.553825    5393 scope.go:122] "RemoveContainer" containerID="1f24b4dd83faac2aafcbd40eacfcb654ff9b23ef8a8216329a4c21c31f3ab741"
Nov 03 10:57:08.554055 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:08.553763    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"
Nov 03 10:57:08.610959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:08.610340    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"]
Nov 03 10:57:08.614432 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:08.614403    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a35a97-openshift-default-5cbc948f7c-pr99f"]
Nov 03 10:57:10.300209 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:10.299794    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" path="/var/lib/kubelet/pods/5f8cc142-76a0-4c09-85bb-e5230e83bdbc/volumes"
Nov 03 10:57:13.195415 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:57:13.195379    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:57:13.352407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.352369    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"]
Nov 03 10:57:13.353639 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.353612    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy"
Nov 03 10:57:13.353820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.353785    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy"
Nov 03 10:57:13.353998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.353835    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy"
Nov 03 10:57:13.353998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.353845    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy"
Nov 03 10:57:13.354127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.354050    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy"
Nov 03 10:57:13.354127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.354069    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8cc142-76a0-4c09-85bb-e5230e83bdbc" containerName="istio-proxy"
Nov 03 10:57:13.360579 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.359857    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.363935 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.363895    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"]
Nov 03 10:57:13.366139 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.366117    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-6012b5\"/\"auth-test-model-6012b5-kserve-self-signed-certs\""
Nov 03 10:57:13.370777 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.369706    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-6012b5\"/\"default-dockercfg-njlsp\""
Nov 03 10:57:13.370777 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.370556    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-6012b5\"/\"kube-root-ca.crt\""
Nov 03 10:57:13.370777 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.370599    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-6012b5\"/\"openshift-service-ca.crt\""
Nov 03 10:57:13.445681 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.445589    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1155f79-030f-4be9-8ce5-5123a4f936c2-tls-certs\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.445864 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.445778    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-model-cache\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.445931 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.445891    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-tmp-dir\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.446002 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.445988    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-home\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.446151 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.446133    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zmqb\" (UniqueName: \"kubernetes.io/projected/e1155f79-030f-4be9-8ce5-5123a4f936c2-kube-api-access-9zmqb\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.446296 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.446269    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-dshm\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.547900    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-model-cache\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.548040    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-tmp-dir\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.548121    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-home\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.548217    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zmqb\" (UniqueName: \"kubernetes.io/projected/e1155f79-030f-4be9-8ce5-5123a4f936c2-kube-api-access-9zmqb\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.548315    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-dshm\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.548466    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1155f79-030f-4be9-8ce5-5123a4f936c2-tls-certs\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.549646    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-home\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.549959    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-model-cache\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.550794 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.550205    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-tmp-dir\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.623400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.622807    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zmqb\" (UniqueName: \"kubernetes.io/projected/e1155f79-030f-4be9-8ce5-5123a4f936c2-kube-api-access-9zmqb\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.623400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.623348    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1155f79-030f-4be9-8ce5-5123a4f936c2-tls-certs\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.624434 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.624382    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-dshm\") pod \"auth-test-model-6012b5-kserve-868ff7649-9bh2f\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:13.682424 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:13.682016    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:14.055524 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:14.055435    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"]
Nov 03 10:57:14.059759 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:57:14.059714    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1155f79_030f_4be9_8ce5_5123a4f936c2.slice/crio-4e858107c55336eeec84d322c577c3df6cbaa090dd991c46111d9782da4c4054 WatchSource:0}: Error finding container 4e858107c55336eeec84d322c577c3df6cbaa090dd991c46111d9782da4c4054: Status 404 returned error can't find the container with id 4e858107c55336eeec84d322c577c3df6cbaa090dd991c46111d9782da4c4054
Nov 03 10:57:14.672864 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:14.672828    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f" event={"ID":"e1155f79-030f-4be9-8ce5-5123a4f936c2","Type":"ContainerStarted","Data":"653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc"}
Nov 03 10:57:14.672864 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:14.672878    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f" event={"ID":"e1155f79-030f-4be9-8ce5-5123a4f936c2","Type":"ContainerStarted","Data":"4e858107c55336eeec84d322c577c3df6cbaa090dd991c46111d9782da4c4054"}
Nov 03 10:57:14.682736 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:14.682708    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:14.691102 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:14.691049    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f" podStartSLOduration=1.69103606 podStartE2EDuration="1.69103606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:14.688730228 +0000 UTC m=+1678.593584672" watchObservedRunningTime="2026-11-03 10:57:14.69103606 +0000 UTC m=+1678.595890517"
Nov 03 10:57:15.072056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.067816    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"]
Nov 03 10:57:15.088734 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.088698    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"]
Nov 03 10:57:15.088943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.088834    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.102158 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.093327    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e11de4-openshift-default-dockercfg-l9lfn\""
Nov 03 10:57:15.241943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.241529    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-envoy\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.241943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.241610    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-data\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.241943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.241637    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-podinfo\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.241943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.241696    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33ed6e4-f8e4-4f29-8187-458a85720103-istiod-ca-cert\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.241943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.241742    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-certs\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.241943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.241816    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqbw\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-kube-api-access-gxqbw\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.241943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.241881    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-token\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.242394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.242027    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.242394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.242060    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-credential-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.350957    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.351017    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-credential-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.351178    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-envoy\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.351244    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-data\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.351268    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-podinfo\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.351316    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33ed6e4-f8e4-4f29-8187-458a85720103-istiod-ca-cert\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.351357    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-certs\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.351410    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqbw\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-kube-api-access-gxqbw\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.351462    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-token\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.352168    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.356531 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.352402    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-credential-socket\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.362964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.361097    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-certs\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.362964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.361620    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33ed6e4-f8e4-4f29-8187-458a85720103-istiod-ca-cert\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.362964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.361897    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-data\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.384043 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.368553    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-envoy\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.415867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.415799    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqbw\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-kube-api-access-gxqbw\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.429643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.429587    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-token\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.451219 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.432518    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-podinfo\") pod \"e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") " pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:15.710896 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:15.710799    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:16.019424 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:57:16.019387    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33ed6e4_f8e4_4f29_8187_458a85720103.slice/crio-ea18298e98dbc986a49e4165364dd9d34380ba93481412be90a3b0e64c7183b4 WatchSource:0}: Error finding container ea18298e98dbc986a49e4165364dd9d34380ba93481412be90a3b0e64c7183b4: Status 404 returned error can't find the container with id ea18298e98dbc986a49e4165364dd9d34380ba93481412be90a3b0e64c7183b4
Nov 03 10:57:16.024122 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:16.024094    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"]
Nov 03 10:57:16.026813 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:16.026771    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:16.026963 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:16.026865    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:16.026963 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:16.026904    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:16.690329 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:16.690290    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" event={"ID":"e33ed6e4-f8e4-4f29-8187-458a85720103","Type":"ContainerStarted","Data":"e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8"}
Nov 03 10:57:16.690552 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:16.690351    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" event={"ID":"e33ed6e4-f8e4-4f29-8187-458a85720103","Type":"ContainerStarted","Data":"ea18298e98dbc986a49e4165364dd9d34380ba93481412be90a3b0e64c7183b4"}
Nov 03 10:57:16.712017 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:16.711959    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" podStartSLOduration=1.711907184 podStartE2EDuration="1.711907184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:16.71060874 +0000 UTC m=+1680.615463219" watchObservedRunningTime="2026-11-03 10:57:16.711907184 +0000 UTC m=+1680.616761624"
Nov 03 10:57:16.714041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:16.714001    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:17.737025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:17.736127    5393 generic.go:356] "Generic (PLEG): container finished" podID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerID="c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4" exitCode=0
Nov 03 10:57:17.737551 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:17.737205    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" event={"ID":"4c7724d8-aa88-49c8-9a08-844b43f5a00d","Type":"ContainerDied","Data":"c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4"}
Nov 03 10:57:17.746735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:17.737673    5393 scope.go:122] "RemoveContainer" containerID="c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4"
Nov 03 10:57:17.746735 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:17.738604    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:17.778945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:17.760679    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:17.779556 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:17.779517    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:17.779761 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:17.779744    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:18.754613 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:18.754581    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" event={"ID":"4c7724d8-aa88-49c8-9a08-844b43f5a00d","Type":"ContainerStarted","Data":"83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab"}
Nov 03 10:57:18.755131 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:18.754960    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:18.756398 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:18.756372    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:57:19.147668 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.147637    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:57:19.148756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.148728    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 10:57:19.148912 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.148855    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 10:57:19.350800 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.350716    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"]
Nov 03 10:57:19.357510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.355793    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.357510 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.356461    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"]
Nov 03 10:57:19.362755 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.362728    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-836589-openshift-default-dockercfg-hw772\""
Nov 03 10:57:19.364820 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.364782    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlr47\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-kube-api-access-rlr47\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.365084 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.365063    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-envoy\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.365210 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.365197    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-podinfo\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.365338 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.365325    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-credential-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.365483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.365469    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.365654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.365639    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istiod-ca-cert\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.365808 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.365792    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-data\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.365988 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.365970    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-token\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.366112 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.366098    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-certs\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.467671 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.467585    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istiod-ca-cert\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.467671 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.467659    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-data\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.467955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.467745    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-token\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.467955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.467784    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-certs\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.467955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.467823    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlr47\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-kube-api-access-rlr47\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.467955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.467877    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-envoy\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.468176 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.467962    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-podinfo\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.468176 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.468044    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-credential-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.468176 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.468116    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.468325 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.468109    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-data\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.468477 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.468452    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istiod-ca-cert\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.468695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.468670    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-certs\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.468759 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.468728    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-credential-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.468865 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.468849    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-socket\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.471352 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.471328    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-envoy\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.475723 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.473219    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-podinfo\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.476841 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.476814    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-token\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.477009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.476871    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlr47\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-kube-api-access-rlr47\") pod \"e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") " pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:19.681630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:19.680791    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:20.147884 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.147851    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 10:57:20.148304 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.147903    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 10:57:20.202378 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.202292    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:20.202549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.202397    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:20.202549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.202441    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:20.211229 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.209121    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"]
Nov 03 10:57:20.844137 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.844090    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:57:20.853577 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.852460    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" event={"ID":"dd4a2f72-5f48-432c-87c1-9a81cd568e12","Type":"ContainerStarted","Data":"6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0"}
Nov 03 10:57:20.853577 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.852524    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" event={"ID":"dd4a2f72-5f48-432c-87c1-9a81cd568e12","Type":"ContainerStarted","Data":"03a9d3e96b99399f90a7d01e55ce7dee943ac2f33fe23b858499d4475376fcdb"}
Nov 03 10:57:20.878286 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:20.878198    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" podStartSLOduration=1.878179865 podStartE2EDuration="1.878179865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:20.874127244 +0000 UTC m=+1684.778981685" watchObservedRunningTime="2026-11-03 10:57:20.878179865 +0000 UTC m=+1684.783034308"
Nov 03 10:57:21.154939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:21.154389    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 10:57:21.154939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:21.154447    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 10:57:21.709944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:21.706251    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:21.804946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:21.804805    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"]
Nov 03 10:57:22.108944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.100794    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:22.108944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.100842    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:22.108944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.100967    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:22.108944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.107712    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"]
Nov 03 10:57:22.156943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.155841    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3ba121-dockercfg-mw486\""
Nov 03 10:57:22.156943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.156797    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3ba121\""
Nov 03 10:57:22.171516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.159597    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b036730-1cf2-4241-9213-68653413a525-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm\" (UID: \"8b036730-1cf2-4241-9213-68653413a525\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:22.171516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.159778    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqd4\" (UniqueName: \"kubernetes.io/projected/8b036730-1cf2-4241-9213-68653413a525-kube-api-access-4rqd4\") pod \"payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm\" (UID: \"8b036730-1cf2-4241-9213-68653413a525\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:22.171516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.169268    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 10:57:22.171516 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.169327    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 10:57:22.270954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.269571    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b036730-1cf2-4241-9213-68653413a525-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm\" (UID: \"8b036730-1cf2-4241-9213-68653413a525\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:22.270954 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.269768    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqd4\" (UniqueName: \"kubernetes.io/projected/8b036730-1cf2-4241-9213-68653413a525-kube-api-access-4rqd4\") pod \"payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm\" (UID: \"8b036730-1cf2-4241-9213-68653413a525\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:22.273947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.271232    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b036730-1cf2-4241-9213-68653413a525-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm\" (UID: \"8b036730-1cf2-4241-9213-68653413a525\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:22.327669 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.327639    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqd4\" (UniqueName: \"kubernetes.io/projected/8b036730-1cf2-4241-9213-68653413a525-kube-api-access-4rqd4\") pod \"payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm\" (UID: \"8b036730-1cf2-4241-9213-68653413a525\") " pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:22.562988 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.562851    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:22.968945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:22.968143    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:57:23.067940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:23.054117    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"]
Nov 03 10:57:23.154291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:23.148233    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 10:57:23.154291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:23.148278    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 10:57:23.686512 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:23.686479    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:23.811467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:23.811436    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:23.969903 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:23.969819    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm" event={"ID":"8b036730-1cf2-4241-9213-68653413a525","Type":"ContainerStarted","Data":"17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb"}
Nov 03 10:57:23.969903 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:23.969878    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm" event={"ID":"8b036730-1cf2-4241-9213-68653413a525","Type":"ContainerStarted","Data":"68952391ba94f63ebded6bcf590ab967db973814e0501f42b8998d1eb1528274"}
Nov 03 10:57:24.009955 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:24.002797    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm" podStartSLOduration=3.002777534 podStartE2EDuration="3.002777534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:24.002441925 +0000 UTC m=+1687.907296361" watchObservedRunningTime="2026-11-03 10:57:24.002777534 +0000 UTC m=+1687.907631976"
Nov 03 10:57:24.014396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:24.013949    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 10:57:24.147363 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:24.147322    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 10:57:24.147534 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:24.147377    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 10:57:24.976733 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:24.976690    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:25.315027 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:25.307167    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:57:25.315027 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:25.309851    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 10:57:31.056334 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.056283    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"]
Nov 03 10:57:31.064169 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.064142    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.075940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.075738    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cc8214\"/\"default-dockercfg-8x489\""
Nov 03 10:57:31.075940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.075828    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cc8214\"/\"auth-test-model-cc8214-kserve-self-signed-certs\""
Nov 03 10:57:31.077388 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.077357    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"]
Nov 03 10:57:31.124738 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.124702    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cc8214\"/\"kube-root-ca.crt\""
Nov 03 10:57:31.124952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.124788    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cc8214\"/\"openshift-service-ca.crt\""
Nov 03 10:57:31.129129 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.129106    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0819cc7-284b-45b2-865a-02e278372617-tls-certs\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.129265 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.129157    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-dshm\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.129373 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.129350    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-tmp-dir\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.129489 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.129468    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-home\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.129583 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.129567    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-model-cache\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.129632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.129607    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhcm5\" (UniqueName: \"kubernetes.io/projected/e0819cc7-284b-45b2-865a-02e278372617-kube-api-access-hhcm5\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.231261 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.231223    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-model-cache\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.232330 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.232304    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhcm5\" (UniqueName: \"kubernetes.io/projected/e0819cc7-284b-45b2-865a-02e278372617-kube-api-access-hhcm5\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.233961 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.233941    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0819cc7-284b-45b2-865a-02e278372617-tls-certs\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.234544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.234526    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-dshm\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.238544 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.235231    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-tmp-dir\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.239537 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.232183    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-model-cache\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.239697 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.239355    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-tmp-dir\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.240018 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.239983    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-home\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.240675 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.240545    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-home\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.266939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.266052    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0819cc7-284b-45b2-865a-02e278372617-tls-certs\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.266939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.266324    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-dshm\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.291959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.284201    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhcm5\" (UniqueName: \"kubernetes.io/projected/e0819cc7-284b-45b2-865a-02e278372617-kube-api-access-hhcm5\") pod \"auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:31.406962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:31.399226    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:32.141492 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:32.141460    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"]
Nov 03 10:57:33.207132 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:33.207077    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k" event={"ID":"e0819cc7-284b-45b2-865a-02e278372617","Type":"ContainerStarted","Data":"4f6bc482a3431d851e65007257865517315c76ad10468ca2c7548159f7cc2858"}
Nov 03 10:57:33.281730 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:33.281685    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k" event={"ID":"e0819cc7-284b-45b2-865a-02e278372617","Type":"ContainerStarted","Data":"2f6f85cdfad8a34188acecd0816eb33fdb5e6f516982bbb9d855158b7dadd69c"}
Nov 03 10:57:33.282477 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:33.260207    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k" podStartSLOduration=2.260184368 podStartE2EDuration="2.260184368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:33.248254304 +0000 UTC m=+1697.153108748" watchObservedRunningTime="2026-11-03 10:57:33.260184368 +0000 UTC m=+1697.165038807"
Nov 03 10:57:33.408062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:33.408025    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:36.037944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:36.037105    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:57:41.400246 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:41.400207    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:41.414092 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:41.414064    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:42.389624 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:42.389601    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 10:57:43.191324 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:57:43.191288    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:57:43.386206 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:43.386171    5393 generic.go:356] "Generic (PLEG): container finished" podID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerID="781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da" exitCode=0
Nov 03 10:57:43.386407 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:43.386248    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" event={"ID":"2e4bbcff-54aa-47e7-ae55-1e451c34a943","Type":"ContainerDied","Data":"781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da"}
Nov 03 10:57:43.386750 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:43.386721    5393 scope.go:122] "RemoveContainer" containerID="781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da"
Nov 03 10:57:43.388437 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:43.388412    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:43.388514 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:43.388480    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:43.388550 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:43.388512    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:57:44.411942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:44.411862    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" event={"ID":"2e4bbcff-54aa-47e7-ae55-1e451c34a943","Type":"ContainerStarted","Data":"988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af"}
Nov 03 10:57:45.210957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:45.208650    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:57:45.210957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:45.208773    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:57:45.210957 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:45.208816    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 10:57:46.222947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:46.213318    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:57:46.222947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:46.213397    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 10:57:47.209320 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.209284    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:57:47.209654 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.209630    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 10:57:47.630019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.626641    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"]
Nov 03 10:57:47.632297 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.632271    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"]
Nov 03 10:57:47.632446 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.632380    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.644934 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.640544    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-47a8a4\"/\"openshift-service-ca.crt\""
Nov 03 10:57:47.644934 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.640658    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-47a8a4\"/\"auth-test-model-47a8a4-kserve-self-signed-certs\""
Nov 03 10:57:47.644934 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.641083    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-47a8a4\"/\"kube-root-ca.crt\""
Nov 03 10:57:47.647677 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.646159    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-47a8a4\"/\"default-dockercfg-t2fjp\""
Nov 03 10:57:47.830945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.830659    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-tmp-dir\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.830945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.830755    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-model-cache\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.830945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.830937    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-home\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.831252 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.830976    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhk7\" (UniqueName: \"kubernetes.io/projected/3f7a8de3-279c-47bc-8086-834be967a412-kube-api-access-9mhk7\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.831252 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.831154    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-dshm\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.831364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.831282    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7a8de3-279c-47bc-8086-834be967a412-tls-certs\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.933783    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-tmp-dir\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.933862    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-model-cache\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.933955    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-home\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.933992    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhk7\" (UniqueName: \"kubernetes.io/projected/3f7a8de3-279c-47bc-8086-834be967a412-kube-api-access-9mhk7\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.934107    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-dshm\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.934218    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7a8de3-279c-47bc-8086-834be967a412-tls-certs\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.935045    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-home\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.935236    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-tmp-dir\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.937425 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.935274    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-model-cache\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.940009 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.939654    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7a8de3-279c-47bc-8086-834be967a412-tls-certs\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.957019 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.956985    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhk7\" (UniqueName: \"kubernetes.io/projected/3f7a8de3-279c-47bc-8086-834be967a412-kube-api-access-9mhk7\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:47.957867 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:47.957228    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-dshm\") pod \"auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") " pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:48.213998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:48.209317    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:57:48.213998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:48.209374    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 10:57:48.252947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:48.252259    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:48.750149 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:48.750103    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"]
Nov 03 10:57:49.210263 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:49.210224    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:57:49.210474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:49.210286    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 10:57:49.520998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:49.520376    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf" event={"ID":"3f7a8de3-279c-47bc-8086-834be967a412","Type":"ContainerStarted","Data":"263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348"}
Nov 03 10:57:49.520998 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:49.520437    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf" event={"ID":"3f7a8de3-279c-47bc-8086-834be967a412","Type":"ContainerStarted","Data":"901f9f9c72d953febaeba275f329013c852aee7f2a2deb0fc50fc6cfd31bff3a"}
Nov 03 10:57:49.542565 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:49.542519    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf" podStartSLOduration=2.542504661 podStartE2EDuration="2.542504661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:57:49.537327665 +0000 UTC m=+1713.442182097" watchObservedRunningTime="2026-11-03 10:57:49.542504661 +0000 UTC m=+1713.447359103"
Nov 03 10:57:50.207668 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:50.207632    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 10:57:50.208248 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:50.207696    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 10:57:50.253154 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:50.253122    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:51.214627 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:51.214590    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:57:51.216205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:51.216182    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:57:51.217166 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:51.217144    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 10:57:58.253526 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:58.253497    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:58.266962 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:58.266897    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:57:58.771941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:57:58.771274    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 10:58:02.133943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.133250    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"]
Nov 03 10:58:02.140943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.140730    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.141305 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.141243    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"]
Nov 03 10:58:02.148548 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.148524    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3ba121\"/\"openshift-service-ca.crt\""
Nov 03 10:58:02.149973 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.149933    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3ba121\"/\"default-dockercfg-n5nqd\""
Nov 03 10:58:02.150056 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.150036    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3ba121\"/\"kube-root-ca.crt\""
Nov 03 10:58:02.150127 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.150109    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3ba121\"/\"explicit-model-3ba121-kserve-self-signed-certs\""
Nov 03 10:58:02.179204 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.179154    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjwc\" (UniqueName: \"kubernetes.io/projected/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-kube-api-access-hmjwc\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.179483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.179455    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tls-certs\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.179619 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.179547    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-model-cache\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.179684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.179652    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tmp-dir\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.179684 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.179678    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-home\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.179839 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.179824    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-dshm\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.280744 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.280717    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjwc\" (UniqueName: \"kubernetes.io/projected/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-kube-api-access-hmjwc\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.281089 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.281069    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tls-certs\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.282631 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.282607    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-model-cache\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.282904 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.282885    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tmp-dir\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.283058 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.283041    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-home\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.283376 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.283354    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-dshm\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.284486 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.284465    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-model-cache\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.287959 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.284839    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tmp-dir\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.288406 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.288386    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-home\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.298464 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.297436    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjwc\" (UniqueName: \"kubernetes.io/projected/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-kube-api-access-hmjwc\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.338944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.336337    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-dshm\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.342249 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.342215    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tls-certs\") pod \"explicit-model-3ba121-kserve-5c9d79d58c-vhbq8\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") " pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:02.484282 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:02.484197    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:03.153830 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:03.153803    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"]
Nov 03 10:58:03.840366 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:03.840271    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8" event={"ID":"67bbf217-0fbb-46a2-b9e5-ed87f92a2942","Type":"ContainerStarted","Data":"be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6"}
Nov 03 10:58:03.840366 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:03.840318    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8" event={"ID":"67bbf217-0fbb-46a2-b9e5-ed87f92a2942","Type":"ContainerStarted","Data":"79a7c75ee0bab87d93bd0a88c05aadf3d8dfa6c50151367c3d16b8ebf9a99a2a"}
Nov 03 10:58:03.859094 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:03.859038    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8" podStartSLOduration=1.8590234319999999 podStartE2EDuration="1.859023432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:58:03.855623724 +0000 UTC m=+1727.760478166" watchObservedRunningTime="2026-11-03 10:58:03.859023432 +0000 UTC m=+1727.763877873"
Nov 03 10:58:04.501943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:04.489010    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:07.258652 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:07.258573    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"]
Nov 03 10:58:09.003086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:09.003048    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:58:09.003522 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:09.003123    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:58:10.920886 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.920853    5393 generic.go:356] "Generic (PLEG): container finished" podID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerID="6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0" exitCode=0
Nov 03 10:58:10.921474 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.920963    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" event={"ID":"dd4a2f72-5f48-432c-87c1-9a81cd568e12","Type":"ContainerDied","Data":"6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0"}
Nov 03 10:58:10.921563 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.921542    5393 scope.go:122] "RemoveContainer" containerID="6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0"
Nov 03 10:58:10.923060 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.923040    5393 generic.go:356] "Generic (PLEG): container finished" podID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerID="e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8" exitCode=0
Nov 03 10:58:10.923155 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.923114    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" event={"ID":"e33ed6e4-f8e4-4f29-8187-458a85720103","Type":"ContainerDied","Data":"e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8"}
Nov 03 10:58:10.923438 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.923411    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:10.923491 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.923480    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:10.923491 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.923487    5393 scope.go:122] "RemoveContainer" containerID="e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8"
Nov 03 10:58:10.923561 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.923517    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:10.925054 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.925028    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:10.925162 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.925107    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:10.925228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:10.925156    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:58:11.946539 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:11.946470    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" event={"ID":"dd4a2f72-5f48-432c-87c1-9a81cd568e12","Type":"ContainerStarted","Data":"32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0"}
Nov 03 10:58:11.951359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:11.951334    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" event={"ID":"e33ed6e4-f8e4-4f29-8187-458a85720103","Type":"ContainerStarted","Data":"1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4"}
Nov 03 10:58:12.685357 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:12.685321    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:58:12.685729 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:12.685705    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 10:58:12.685932 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:12.685888    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 10:58:12.713332 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:12.713297    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:58:12.713719 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:12.713623    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 10:58:12.713907 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:12.713885    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 10:58:13.240163 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:58:13.240122    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:58:13.690060 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:13.690019    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 10:58:13.690361 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:13.690334    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 10:58:13.718945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:13.715392    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 10:58:13.718945 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:13.715453    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 10:58:14.698944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:14.697179    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 10:58:14.698944 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:14.697255    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 10:58:14.714552 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:14.713035    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 10:58:14.714552 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:14.713114    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 10:58:15.697109 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:15.697074    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 10:58:15.697437 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:15.697408    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 10:58:15.714554 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:15.714521    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 10:58:15.715237 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:15.715205    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 10:58:16.684946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:16.681308    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 10:58:16.684946 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:16.681386    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 10:58:16.714943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:16.714360    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 10:58:16.714943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:16.714419    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 10:58:17.672093 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:17.672037    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"]
Nov 03 10:58:17.783942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:17.779295    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:58:17.783942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:17.780705    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:58:17.783942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:17.782179    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:58:17.793192 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:17.793156    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 10:58:17.793793 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:17.793773    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:58:17.804379 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:17.803871    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 10:58:22.299470 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.299421    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8" podUID="67bbf217-0fbb-46a2-b9e5-ed87f92a2942" containerName="main" containerID="cri-o://be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6" gracePeriod=45
Nov 03 10:58:22.646986 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.646962    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:22.673816 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.673770    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm" podUID="8b036730-1cf2-4241-9213-68653413a525" containerName="payload-processing" containerID="cri-o://17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb" gracePeriod=25
Nov 03 10:58:22.743135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743102    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-dshm\") pod \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") "
Nov 03 10:58:22.743312 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743206    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-home\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-home\") pod \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") "
Nov 03 10:58:22.743371 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743311    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tls-certs\" (UniqueName: \"kubernetes.io/secret/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tls-certs\") pod \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") "
Nov 03 10:58:22.743371 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743361    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tmp-dir\") pod \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") "
Nov 03 10:58:22.743504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743477    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-home" pod "67bbf217-0fbb-46a2-b9e5-ed87f92a2942" (UID: "67bbf217-0fbb-46a2-b9e5-ed87f92a2942"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:22.743504 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743495    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-model-cache\") pod \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") "
Nov 03 10:58:22.743618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743585    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-kube-api-access-hmjwc\" (UniqueName: \"kubernetes.io/projected/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-kube-api-access-hmjwc\") pod \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\" (UID: \"67bbf217-0fbb-46a2-b9e5-ed87f92a2942\") "
Nov 03 10:58:22.743756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743736    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-model-cache" pod "67bbf217-0fbb-46a2-b9e5-ed87f92a2942" (UID: "67bbf217-0fbb-46a2-b9e5-ed87f92a2942"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:22.743756 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.743679    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tmp-dir" pod "67bbf217-0fbb-46a2-b9e5-ed87f92a2942" (UID: "67bbf217-0fbb-46a2-b9e5-ed87f92a2942"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:22.744199 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.744177    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:22.744325 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.744202    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:22.744325 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.744217    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:22.745777 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.745750    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tls-certs" pod "67bbf217-0fbb-46a2-b9e5-ed87f92a2942" (UID: "67bbf217-0fbb-46a2-b9e5-ed87f92a2942"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:58:22.746133 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.746106    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-dshm" pod "67bbf217-0fbb-46a2-b9e5-ed87f92a2942" (UID: "67bbf217-0fbb-46a2-b9e5-ed87f92a2942"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:58:22.747048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.747019    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-kube-api-access-hmjwc" pod "67bbf217-0fbb-46a2-b9e5-ed87f92a2942" (UID: "67bbf217-0fbb-46a2-b9e5-ed87f92a2942"). InnerVolumeSpecName "kube-api-access-hmjwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:58:22.845710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.845674    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:22.845710 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.845710    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmjwc\" (UniqueName: \"kubernetes.io/projected/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-kube-api-access-hmjwc\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:22.845949 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.845724    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bbf217-0fbb-46a2-b9e5-ed87f92a2942-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:22.959347 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:22.959310    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:58:23.048013 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.047986    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b036730-1cf2-4241-9213-68653413a525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b036730-1cf2-4241-9213-68653413a525-plugins-config-volume\") pod \"8b036730-1cf2-4241-9213-68653413a525\" (UID: \"8b036730-1cf2-4241-9213-68653413a525\") "
Nov 03 10:58:23.048205 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.048029    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b036730-1cf2-4241-9213-68653413a525-kube-api-access-4rqd4\" (UniqueName: \"kubernetes.io/projected/8b036730-1cf2-4241-9213-68653413a525-kube-api-access-4rqd4\") pod \"8b036730-1cf2-4241-9213-68653413a525\" (UID: \"8b036730-1cf2-4241-9213-68653413a525\") "
Nov 03 10:58:23.048417 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.048393    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b036730-1cf2-4241-9213-68653413a525-plugins-config-volume" pod "8b036730-1cf2-4241-9213-68653413a525" (UID: "8b036730-1cf2-4241-9213-68653413a525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:58:23.048623 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.048602    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b036730-1cf2-4241-9213-68653413a525-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:23.050331 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.050295    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b036730-1cf2-4241-9213-68653413a525-kube-api-access-4rqd4" pod "8b036730-1cf2-4241-9213-68653413a525" (UID: "8b036730-1cf2-4241-9213-68653413a525"). InnerVolumeSpecName "kube-api-access-4rqd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:58:23.149004 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.148909    5393 generic.go:356] "Generic (PLEG): container finished" podID="67bbf217-0fbb-46a2-b9e5-ed87f92a2942" containerID="be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6" exitCode=0
Nov 03 10:58:23.149218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.149128    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8" event={"ID":"67bbf217-0fbb-46a2-b9e5-ed87f92a2942","Type":"ContainerDied","Data":"be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6"}
Nov 03 10:58:23.149218 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.149213    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8" event={"ID":"67bbf217-0fbb-46a2-b9e5-ed87f92a2942","Type":"ContainerDied","Data":"79a7c75ee0bab87d93bd0a88c05aadf3d8dfa6c50151367c3d16b8ebf9a99a2a"}
Nov 03 10:58:23.149362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.149241    5393 scope.go:122] "RemoveContainer" containerID="be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6"
Nov 03 10:58:23.149486 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.149444    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"
Nov 03 10:58:23.169435 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.164564    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqd4\" (UniqueName: \"kubernetes.io/projected/8b036730-1cf2-4241-9213-68653413a525-kube-api-access-4rqd4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:58:23.174041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.170413    5393 generic.go:356] "Generic (PLEG): container finished" podID="8b036730-1cf2-4241-9213-68653413a525" containerID="17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb" exitCode=0
Nov 03 10:58:23.174041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.170486    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm" event={"ID":"8b036730-1cf2-4241-9213-68653413a525","Type":"ContainerDied","Data":"17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb"}
Nov 03 10:58:23.174041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.170535    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm" event={"ID":"8b036730-1cf2-4241-9213-68653413a525","Type":"ContainerDied","Data":"68952391ba94f63ebded6bcf590ab967db973814e0501f42b8998d1eb1528274"}
Nov 03 10:58:23.174041 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.170542    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"
Nov 03 10:58:23.186524 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.186499    5393 scope.go:122] "RemoveContainer" containerID="be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6"
Nov 03 10:58:23.187012 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:58:23.186984    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6\": container with ID starting with be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6 not found: ID does not exist" containerID="be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6"
Nov 03 10:58:23.187110 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.187025    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6"} err="failed to get container status \"be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6\": rpc error: code = NotFound desc = could not find container \"be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6\": container with ID starting with be133a343da8c23b094ca72ac144b84e84cb08f0262bcf9fd25fd71b207a05f6 not found: ID does not exist"
Nov 03 10:58:23.187110 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.187049    5393 scope.go:122] "RemoveContainer" containerID="17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb"
Nov 03 10:58:23.192682 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.192659    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"]
Nov 03 10:58:23.200567 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.200545    5393 scope.go:122] "RemoveContainer" containerID="17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb"
Nov 03 10:58:23.200902 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:58:23.200880    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb\": container with ID starting with 17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb not found: ID does not exist" containerID="17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb"
Nov 03 10:58:23.201069 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.200908    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb"} err="failed to get container status \"17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb\": rpc error: code = NotFound desc = could not find container \"17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb\": container with ID starting with 17fb95e8109bb5fc3a3d48be709c078b48f245c443222eea68ca9349989336fb not found: ID does not exist"
Nov 03 10:58:23.201195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.201170    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-3ba121/explicit-model-3ba121-kserve-5c9d79d58c-vhbq8"]
Nov 03 10:58:23.213444 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.213374    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"]
Nov 03 10:58:23.216630 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:23.216606    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3ba121-6b4d4b46df-8rfwm"]
Nov 03 10:58:24.266803 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.266746    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67bbf217-0fbb-46a2-b9e5-ed87f92a2942" path="/var/lib/kubelet/pods/67bbf217-0fbb-46a2-b9e5-ed87f92a2942/volumes"
Nov 03 10:58:24.268046 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.268023    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b036730-1cf2-4241-9213-68653413a525" path="/var/lib/kubelet/pods/8b036730-1cf2-4241-9213-68653413a525/volumes"
Nov 03 10:58:24.268599 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.268582    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"]
Nov 03 10:58:24.269657 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.269637    5393 cpu_manager.go:414] "containerMap: removing container" podUID="8b036730-1cf2-4241-9213-68653413a525" containerName="payload-processing"
Nov 03 10:58:24.269803 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.269789    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b036730-1cf2-4241-9213-68653413a525" containerName="payload-processing"
Nov 03 10:58:24.269971 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.269957    5393 cpu_manager.go:414] "containerMap: removing container" podUID="67bbf217-0fbb-46a2-b9e5-ed87f92a2942" containerName="main"
Nov 03 10:58:24.270075 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.270061    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67bbf217-0fbb-46a2-b9e5-ed87f92a2942" containerName="main"
Nov 03 10:58:24.270369 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.270356    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b036730-1cf2-4241-9213-68653413a525" containerName="payload-processing"
Nov 03 10:58:24.270472 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.270462    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="67bbf217-0fbb-46a2-b9e5-ed87f92a2942" containerName="main"
Nov 03 10:58:24.274396 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.274380    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"]
Nov 03 10:58:24.274680 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.274640    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.282964 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.282942    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"openshift-service-ca.crt\""
Nov 03 10:58:24.283114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.283096    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"default-dockercfg-xcnf9\""
Nov 03 10:58:24.283180 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.283096    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"auth-test-model-ce5555-kserve-self-signed-certs\""
Nov 03 10:58:24.283243 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.283177    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce5555\"/\"kube-root-ca.crt\""
Nov 03 10:58:24.364941 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.348848    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-model-cache\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.365262 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.365233    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-dshm\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.365460 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.365441    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b69c7ad-8e31-42c8-922d-b9dab996e617-tls-certs\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.365786 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.365746    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-home\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.373942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.366062    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-tmp-dir\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.379940 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.374247    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxwm\" (UniqueName: \"kubernetes.io/projected/1b69c7ad-8e31-42c8-922d-b9dab996e617-kube-api-access-ctxwm\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.487958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.485501    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxwm\" (UniqueName: \"kubernetes.io/projected/1b69c7ad-8e31-42c8-922d-b9dab996e617-kube-api-access-ctxwm\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.487958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.486031    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-model-cache\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.487958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.486098    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-dshm\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.487958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.486196    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b69c7ad-8e31-42c8-922d-b9dab996e617-tls-certs\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.487958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.487056    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-model-cache\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.487958 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.487257    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-home\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.497977 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.497800    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-tmp-dir\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.498291 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.498265    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-tmp-dir\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.501228 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.500709    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-home\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.501975 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.501935    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b69c7ad-8e31-42c8-922d-b9dab996e617-tls-certs\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.543031 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.520042    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-dshm\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.543031 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.521104    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxwm\" (UniqueName: \"kubernetes.io/projected/1b69c7ad-8e31-42c8-922d-b9dab996e617-kube-api-access-ctxwm\") pod \"auth-test-model-ce5555-kserve-5744d85cb9-trs2w\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") " pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.601187 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.601153    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:24.894135 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:24.894104    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"]
Nov 03 10:58:24.895102 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:58:24.895075    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b69c7ad_8e31_42c8_922d_b9dab996e617.slice/crio-a5b129db78112600cd637f8cb04a97e70f326ae734d97246dd11d7630cbd1f4f WatchSource:0}: Error finding container a5b129db78112600cd637f8cb04a97e70f326ae734d97246dd11d7630cbd1f4f: Status 404 returned error can't find the container with id a5b129db78112600cd637f8cb04a97e70f326ae734d97246dd11d7630cbd1f4f
Nov 03 10:58:25.278362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:25.278271    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w" event={"ID":"1b69c7ad-8e31-42c8-922d-b9dab996e617","Type":"ContainerStarted","Data":"828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142"}
Nov 03 10:58:25.278362 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:25.278310    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w" event={"ID":"1b69c7ad-8e31-42c8-922d-b9dab996e617","Type":"ContainerStarted","Data":"a5b129db78112600cd637f8cb04a97e70f326ae734d97246dd11d7630cbd1f4f"}
Nov 03 10:58:25.296235 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:25.296177    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w" podStartSLOduration=1.296142372 podStartE2EDuration="1.296142372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:58:25.29357084 +0000 UTC m=+1749.198425272" watchObservedRunningTime="2026-11-03 10:58:25.296142372 +0000 UTC m=+1749.200996814"
Nov 03 10:58:25.602375 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:25.602342    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:34.602245 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:34.602206    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:34.616467 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:34.616440    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:35.428086 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:35.428057    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 10:58:43.191671 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:58:43.191629    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:58:58.990264 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:58.990231    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"]
Nov 03 10:58:58.994406 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:58.994377    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.007364 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.007336    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"openshift-service-ca.crt\""
Nov 03 10:58:59.010359 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.010337    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"kube-root-ca.crt\""
Nov 03 10:58:59.011974 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.011937    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"default-dockercfg-mclmp\""
Nov 03 10:58:59.012096 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.012049    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"auth-test-model-836589-kserve-self-signed-certs\""
Nov 03 10:58:59.018114 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.018090    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"]
Nov 03 10:58:59.058413 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.058372    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"]
Nov 03 10:58:59.081179 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.081147    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d179392-v5lgg"]
Nov 03 10:58:59.090288 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.090259    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqszh\" (UniqueName: \"kubernetes.io/projected/877bd7e3-b64c-45e0-a339-2a0574e99bef-kube-api-access-zqszh\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.090483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.090313    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-home\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.090483 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.090420    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-dshm\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.090597 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.090573    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-model-cache\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.090656 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.090639    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-tmp-dir\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.090782 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.090765    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877bd7e3-b64c-45e0-a339-2a0574e99bef-tls-certs\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.191894 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.191851    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqszh\" (UniqueName: \"kubernetes.io/projected/877bd7e3-b64c-45e0-a339-2a0574e99bef-kube-api-access-zqszh\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.192141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.191937    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-home\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.192141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.191980    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-dshm\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.192141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.192083    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-model-cache\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.192141 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.192135    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-tmp-dir\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.192372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.192218    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877bd7e3-b64c-45e0-a339-2a0574e99bef-tls-certs\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.192432 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.192394    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-home\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.192620 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.192598    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-model-cache\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.192706 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.192661    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-tmp-dir\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.194723 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.194695    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-dshm\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.195024 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.195007    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877bd7e3-b64c-45e0-a339-2a0574e99bef-tls-certs\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.202811 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.202786    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqszh\" (UniqueName: \"kubernetes.io/projected/877bd7e3-b64c-45e0-a339-2a0574e99bef-kube-api-access-zqszh\") pod \"auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") " pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.315225 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.314710    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:58:59.592538 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.592435    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"]
Nov 03 10:58:59.601403 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.601376    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"]
Nov 03 10:58:59.601728 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.601696    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.611453 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.611424    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-47a8a4-openshift-default-dockercfg-97wwj\""
Nov 03 10:58:59.709711 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.709669    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.709947 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.709724    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xblh9\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-kube-api-access-xblh9\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.710028 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.709940    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.710080 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.710065    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.710138 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.710119    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.710195 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.710178    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.710262 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.710246    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.710301 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.710275    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c280ce5-c5dd-421d-afdd-977f78ced608-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.710346 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.710303    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.749629 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.749601    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"]
Nov 03 10:58:59.750912 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:58:59.750871    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod877bd7e3_b64c_45e0_a339_2a0574e99bef.slice/crio-5dc1a8e5fcf5e418263ec10e3e0cc4ec7a587603a88d50552ef1f762705c3e49 WatchSource:0}: Error finding container 5dc1a8e5fcf5e418263ec10e3e0cc4ec7a587603a88d50552ef1f762705c3e49: Status 404 returned error can't find the container with id 5dc1a8e5fcf5e418263ec10e3e0cc4ec7a587603a88d50552ef1f762705c3e49
Nov 03 10:58:59.811540 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811506    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.811696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811547    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xblh9\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-kube-api-access-xblh9\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.811696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811646    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.811835 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811724    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.811835 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811766    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.811835 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811816    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.812254 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811880    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.812254 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811879    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-credential-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.812254 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.811903    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c280ce5-c5dd-421d-afdd-977f78ced608-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.812254 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.812012    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.812758 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.812729    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-socket\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.812980 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.812905    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j" event={"ID":"877bd7e3-b64c-45e0-a339-2a0574e99bef","Type":"ContainerStarted","Data":"5dc1a8e5fcf5e418263ec10e3e0cc4ec7a587603a88d50552ef1f762705c3e49"}
Nov 03 10:58:59.813111 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.813096    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c280ce5-c5dd-421d-afdd-977f78ced608-istiod-ca-cert\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.813287 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.813205    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-data\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.813400 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.813379    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-certs\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.815250 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.815214    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-envoy\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.815871 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.815852    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-podinfo\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.823518 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.823486    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-token\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.823611 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.823568    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xblh9\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-kube-api-access-xblh9\") pod \"e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") " pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:58:59.936942 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:58:59.932983    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:59:00.375953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:00.370875    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:00.375953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:00.371043    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:00.375953 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:00.371101    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:00.567943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:00.567263    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00349a3-007d-45f0-a826-85ee0e5a1742" path="/var/lib/kubelet/pods/e00349a3-007d-45f0-a826-85ee0e5a1742/volumes"
Nov 03 10:59:00.567943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:00.567718    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"]
Nov 03 10:59:00.856528 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:00.856368    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j" event={"ID":"877bd7e3-b64c-45e0-a339-2a0574e99bef","Type":"ContainerStarted","Data":"b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181"}
Nov 03 10:59:00.884549 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:00.859910    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" event={"ID":"7c280ce5-c5dd-421d-afdd-977f78ced608","Type":"ContainerStarted","Data":"ce60fdd852c463c5093013c740ba54df3e59138d953c34a1a564a745fc150549"}
Nov 03 10:59:00.900956 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:00.899219    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j" podStartSLOduration=2.899179146 podStartE2EDuration="2.899179146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:00.885538627 +0000 UTC m=+1784.790393065" watchObservedRunningTime="2026-11-03 10:59:00.899179146 +0000 UTC m=+1784.804033588"
Nov 03 10:59:01.320001 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:01.319138    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:59:01.869928 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:01.869887    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" event={"ID":"7c280ce5-c5dd-421d-afdd-977f78ced608","Type":"ContainerStarted","Data":"0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a"}
Nov 03 10:59:01.907628 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:01.907558    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" podStartSLOduration=2.9075348659999998 podStartE2EDuration="2.907534866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:01.898573844 +0000 UTC m=+1785.803428348" watchObservedRunningTime="2026-11-03 10:59:01.907534866 +0000 UTC m=+1785.812389309"
Nov 03 10:59:01.934415 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:01.934384    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:59:01.935883 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:01.935853    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 03 10:59:01.936016 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:01.935966    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 03 10:59:02.934260 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:02.934217    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 03 10:59:02.934768 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:02.934293    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 03 10:59:03.943823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:03.943315    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 03 10:59:03.943823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:03.943401    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 03 10:59:04.933818 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:04.933785    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 03 10:59:04.934076 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:04.933844    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 03 10:59:05.934347 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:05.934316    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 03 10:59:05.934859 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:05.934367    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 03 10:59:06.935943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:06.933795    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 03 10:59:06.935943 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:06.933860    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 03 10:59:07.940861 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:07.940829    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:59:07.941506 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:07.941486    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:59:07.942274 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:07.942253    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 10:59:09.317076 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:09.316617    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:59:09.334177 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:09.334143    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:59:10.034854 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:10.034819    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 10:59:12.021459 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:12.021421    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"]
Nov 03 10:59:12.025064 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:12.025039    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6036a896-7cj5g"]
Nov 03 10:59:12.274786 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:12.274659    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61935e03-223a-410e-97bd-0bdebdb1b7e4" path="/var/lib/kubelet/pods/61935e03-223a-410e-97bd-0bdebdb1b7e4/volumes"
Nov 03 10:59:13.190331 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:59:13.190293    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 10:59:17.403113 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:17.403082    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:59:17.403706 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:17.403420    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 10:59:18.153334 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:18.153304    5393 scope.go:122] "RemoveContainer" containerID="f97076ad29cb5d625274bd53d9e37c979c6ba3fa09d62e927c5db312c0059e60"
Nov 03 10:59:28.846416 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:28.846367    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"]
Nov 03 10:59:28.847259 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:28.847224    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy" containerID="cri-o://7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14" gracePeriod=30
Nov 03 10:59:29.194102 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.194028    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"]
Nov 03 10:59:29.200566 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.200545    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.204993 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.204969    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e92a08-openshift-default-dockercfg-j59b6\""
Nov 03 10:59:29.210723 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.210696    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"]
Nov 03 10:59:29.277108 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277068    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e670692d-bbc2-4941-b905-a68856f100b4-istio-podinfo\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.277108 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277104    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-certs\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.277372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277215    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-credential-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.277372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277271    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e670692d-bbc2-4941-b905-a68856f100b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.277372 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277364    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9mcp\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-kube-api-access-j9mcp\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.277532 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277413    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-data\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.277598 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277580    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.277649 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277638    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-istio-token\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.277748 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.277677    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-envoy\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.378799 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.378757    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-istio-token\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.378833    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-envoy\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.378902    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e670692d-bbc2-4941-b905-a68856f100b4-istio-podinfo\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.378952    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-certs\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379062 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.379018    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-credential-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.379069    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e670692d-bbc2-4941-b905-a68856f100b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.379136    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9mcp\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-kube-api-access-j9mcp\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379328 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.379197    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-data\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.379349    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.379490 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.379422    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-certs\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.380025 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.379974    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-data\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.380175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.380061    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-credential-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.380175 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.380163    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e670692d-bbc2-4941-b905-a68856f100b4-istiod-ca-cert\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.380463 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.380440    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-socket\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.381740 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.381717    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-envoy\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.382457 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.382437    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e670692d-bbc2-4941-b905-a68856f100b4-istio-podinfo\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.386752 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.386725    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-istio-token\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.387695 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.387670    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9mcp\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-kube-api-access-j9mcp\") pod \"e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") " pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.518829 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.518735    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:29.803844 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.803817    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"]
Nov 03 10:59:29.805075 ip-10-0-134-12 kubenswrapper[5393]: W1103 10:59:29.805050    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode670692d_bbc2_4941_b905_a68856f100b4.slice/crio-34324785ecaf2a42d30232b1ea380490357cb695f8a28389817a00fafe6fd22f WatchSource:0}: Error finding container 34324785ecaf2a42d30232b1ea380490357cb695f8a28389817a00fafe6fd22f: Status 404 returned error can't find the container with id 34324785ecaf2a42d30232b1ea380490357cb695f8a28389817a00fafe6fd22f
Nov 03 10:59:29.807622 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.807592    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:29.807717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.807653    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:29.807717 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:29.807681    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:59:30.339939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:30.338870    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" event={"ID":"e670692d-bbc2-4941-b905-a68856f100b4","Type":"ContainerStarted","Data":"ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d"}
Nov 03 10:59:30.339939 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:30.338967    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" event={"ID":"e670692d-bbc2-4941-b905-a68856f100b4","Type":"ContainerStarted","Data":"34324785ecaf2a42d30232b1ea380490357cb695f8a28389817a00fafe6fd22f"}
Nov 03 10:59:30.395952 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:30.389052    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podStartSLOduration=1.3890268940000001 podStartE2EDuration="1.389026894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:59:30.364908127 +0000 UTC m=+1814.269762568" watchObservedRunningTime="2026-11-03 10:59:30.389026894 +0000 UTC m=+1814.293881351"
Nov 03 10:59:30.519083 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:30.519046    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:30.521140 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:30.521112    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 03 10:59:30.521309 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:30.521159    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 03 10:59:31.519459 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:31.519424    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 03 10:59:31.519869 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:31.519483    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 03 10:59:32.519373 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:32.519337    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 03 10:59:32.519557 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:32.519394    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 03 10:59:33.519278 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:33.519244    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 03 10:59:33.519484 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:33.519303    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 03 10:59:34.180487 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.180464    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:59:34.273374 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273327    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-envoy\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.273587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273387    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-istio-token\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-istio-token\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.273587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273422    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fe9672-f20f-4add-965c-e93aa9492094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe9672-f20f-4add-965c-e93aa9492094-istiod-ca-cert\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.273587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273470    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-certs\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.273587 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273504    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-kube-api-access-p7tp8\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-kube-api-access-p7tp8\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.273824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273634    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-socket\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.273824 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273800    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69fe9672-f20f-4add-965c-e93aa9492094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe9672-f20f-4add-965c-e93aa9492094-istio-podinfo\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.273996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273853    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-credential-socket\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.273996 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.273982    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-data\") pod \"69fe9672-f20f-4add-965c-e93aa9492094\" (UID: \"69fe9672-f20f-4add-965c-e93aa9492094\") "
Nov 03 10:59:34.274632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274224    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-socket" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:59:34.274632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274380    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-credential-socket" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:59:34.274632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274427    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-data" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:59:34.274632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274576    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fe9672-f20f-4add-965c-e93aa9492094-istiod-ca-cert" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:59:34.274632 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274606    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-certs" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:59:34.275020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274744    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.275020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274762    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.275020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274776    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.275020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274790    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.275020 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.274805    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe9672-f20f-4add-965c-e93aa9492094-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.277072 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.277041    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69fe9672-f20f-4add-965c-e93aa9492094-istio-podinfo" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:59:34.277394 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.277369    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-istio-token" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:59:34.277511 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.277405    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-kube-api-access-p7tp8" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "kube-api-access-p7tp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:59:34.278048 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.278020    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-envoy" pod "69fe9672-f20f-4add-965c-e93aa9492094" (UID: "69fe9672-f20f-4add-965c-e93aa9492094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:59:34.366823 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.366782    5393 generic.go:356] "Generic (PLEG): container finished" podID="69fe9672-f20f-4add-965c-e93aa9492094" containerID="7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14" exitCode=0
Nov 03 10:59:34.367059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.366832    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"
Nov 03 10:59:34.367059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.366868    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" event={"ID":"69fe9672-f20f-4add-965c-e93aa9492094","Type":"ContainerDied","Data":"7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14"}
Nov 03 10:59:34.367059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.366957    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv" event={"ID":"69fe9672-f20f-4add-965c-e93aa9492094","Type":"ContainerDied","Data":"7e54b05cf58fd7877a1bc4b87faa77e94ee5903907622ffd9c35261aeab8b3ea"}
Nov 03 10:59:34.367059 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.366990    5393 scope.go:122] "RemoveContainer" containerID="7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14"
Nov 03 10:59:34.376508 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.376485    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe9672-f20f-4add-965c-e93aa9492094-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.376614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.376509    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe9672-f20f-4add-965c-e93aa9492094-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.376614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.376523    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.376614 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.376533    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7tp8\" (UniqueName: \"kubernetes.io/projected/69fe9672-f20f-4add-965c-e93aa9492094-kube-api-access-p7tp8\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 10:59:34.380310 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.380292    5393 scope.go:122] "RemoveContainer" containerID="7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14"
Nov 03 10:59:34.380621 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:59:34.380601    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14\": container with ID starting with 7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14 not found: ID does not exist" containerID="7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14"
Nov 03 10:59:34.380668 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.380629    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14"} err="failed to get container status \"7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14\": rpc error: code = NotFound desc = could not find container \"7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14\": container with ID starting with 7222074db6687a1a59ee778682715279fcd876f016b6e2e1bc525b3a2c6d7a14 not found: ID does not exist"
Nov 03 10:59:34.413091 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.412996    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"]
Nov 03 10:59:34.416643 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.416612    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-qm5hv"]
Nov 03 10:59:34.519469 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.519435    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 03 10:59:34.519642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:34.519490    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 03 10:59:35.519861 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:35.519819    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 03 10:59:35.520275 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:35.519878    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 03 10:59:36.234642 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:36.234611    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fe9672-f20f-4add-965c-e93aa9492094" path="/var/lib/kubelet/pods/69fe9672-f20f-4add-965c-e93aa9492094/volumes"
Nov 03 10:59:36.542202 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:36.542101    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:36.542696 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:36.542563    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:36.543618 ip-10-0-134-12 kubenswrapper[5393]: I1103 10:59:36.543596    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 10:59:43.193005 ip-10-0-134-12 kubenswrapper[5393]: E1103 10:59:43.192958    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:00:00.142847 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.142759    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"]
Nov 03 11:00:00.143686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.143664    5393 cpu_manager.go:414] "containerMap: removing container" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy"
Nov 03 11:00:00.143793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.143688    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy"
Nov 03 11:00:00.143891 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.143877    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fe9672-f20f-4add-965c-e93aa9492094" containerName="istio-proxy"
Nov 03 11:00:00.147812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.147787    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"]
Nov 03 11:00:00.148012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.147945    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:00.153060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.153035    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:00.161815 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.161782    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"]
Nov 03 11:00:00.209972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.205256    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"]
Nov 03 11:00:00.209972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.205342    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"]
Nov 03 11:00:00.209972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.205728    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:00.218610 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.218577    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"]
Nov 03 11:00:00.218806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.218628    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"]
Nov 03 11:00:00.218889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.218816    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:00.227980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.227792    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:00.242430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.242392    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"]
Nov 03 11:00:00.242617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.242444    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"]
Nov 03 11:00:00.242617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.242470    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"]
Nov 03 11:00:00.242617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.242494    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"]
Nov 03 11:00:00.246991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.246972    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"]
Nov 03 11:00:00.247413 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.247390    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:00.251039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.251020    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:00:00.254661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.254641    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"]
Nov 03 11:00:00.255022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.255001    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:00.260283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.259987    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"]
Nov 03 11:00:00.264566 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.264537    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"]
Nov 03 11:00:00.269715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.269688    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"]
Nov 03 11:00:00.269872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.269836    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:00.319782 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.319745    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be96e79-cf81-4f39-a1e3-b987253db254-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k\" (UID: \"3be96e79-cf81-4f39-a1e3-b987253db254\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:00.319996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.319788    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5b96\" (UniqueName: \"kubernetes.io/projected/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-kube-api-access-q5b96\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn\" (UID: \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:00.319996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.319871    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xdv\" (UniqueName: \"kubernetes.io/projected/3be96e79-cf81-4f39-a1e3-b987253db254-kube-api-access-l6xdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k\" (UID: \"3be96e79-cf81-4f39-a1e3-b987253db254\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:00.320102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320003    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm\" (UID: \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:00.320102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320023    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrdf\" (UniqueName: \"kubernetes.io/projected/e03f8fd9-8386-43a9-9a6f-06e28c66e515-kube-api-access-lsrdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9\" (UID: \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:00.320102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320041    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nht\" (UniqueName: \"kubernetes.io/projected/e29b9634-c42c-4209-9ad5-5475064197de-kube-api-access-x5nht\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42\" (UID: \"e29b9634-c42c-4209-9ad5-5475064197de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:00.320246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320148    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9cr\" (UniqueName: \"kubernetes.io/projected/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-kube-api-access-xn9cr\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6\" (UID: \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:00.320246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320217    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29b9634-c42c-4209-9ad5-5475064197de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42\" (UID: \"e29b9634-c42c-4209-9ad5-5475064197de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:00.320362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320336    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7swp\" (UniqueName: \"kubernetes.io/projected/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-kube-api-access-w7swp\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm\" (UID: \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:00.320466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320450    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn\" (UID: \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:00.320531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320490    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f8fd9-8386-43a9-9a6f-06e28c66e515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9\" (UID: \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:00.320581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.320540    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6\" (UID: \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:00.429989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.427644    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm\" (UID: \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:00.429989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.427773    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrdf\" (UniqueName: \"kubernetes.io/projected/e03f8fd9-8386-43a9-9a6f-06e28c66e515-kube-api-access-lsrdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9\" (UID: \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:00.429989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.429040    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm\" (UID: \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:00.429989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.429470    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nht\" (UniqueName: \"kubernetes.io/projected/e29b9634-c42c-4209-9ad5-5475064197de-kube-api-access-x5nht\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42\" (UID: \"e29b9634-c42c-4209-9ad5-5475064197de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:00.429989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.429584    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676e924d-1a19-405c-a7d9-0489ccd9dccd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v\" (UID: \"676e924d-1a19-405c-a7d9-0489ccd9dccd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:00.430692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.429803    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9cr\" (UniqueName: \"kubernetes.io/projected/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-kube-api-access-xn9cr\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6\" (UID: \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:00.430692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430255    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a5651-9488-4a2e-bec3-c1a3d3e94069-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t\" (UID: \"493a5651-9488-4a2e-bec3-c1a3d3e94069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:00.430692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430356    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29b9634-c42c-4209-9ad5-5475064197de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42\" (UID: \"e29b9634-c42c-4209-9ad5-5475064197de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:00.430692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430422    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7swp\" (UniqueName: \"kubernetes.io/projected/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-kube-api-access-w7swp\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm\" (UID: \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:00.430692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430467    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9s4z\" (UniqueName: \"kubernetes.io/projected/493a5651-9488-4a2e-bec3-c1a3d3e94069-kube-api-access-x9s4z\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t\" (UID: \"493a5651-9488-4a2e-bec3-c1a3d3e94069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:00.430692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430632    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn\" (UID: \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:00.430692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430671    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f8fd9-8386-43a9-9a6f-06e28c66e515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9\" (UID: \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:00.431098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430742    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6\" (UID: \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:00.431098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430780    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be96e79-cf81-4f39-a1e3-b987253db254-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k\" (UID: \"3be96e79-cf81-4f39-a1e3-b987253db254\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:00.431098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430813    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5b96\" (UniqueName: \"kubernetes.io/projected/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-kube-api-access-q5b96\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn\" (UID: \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:00.431098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.430856    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xdv\" (UniqueName: \"kubernetes.io/projected/3be96e79-cf81-4f39-a1e3-b987253db254-kube-api-access-l6xdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k\" (UID: \"3be96e79-cf81-4f39-a1e3-b987253db254\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:00.441421 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.439814    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn\" (UID: \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:00.441421 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.440940    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29b9634-c42c-4209-9ad5-5475064197de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42\" (UID: \"e29b9634-c42c-4209-9ad5-5475064197de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:00.447100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.442632    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f8fd9-8386-43a9-9a6f-06e28c66e515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9\" (UID: \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:00.447100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.447027    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzq8\" (UniqueName: \"kubernetes.io/projected/676e924d-1a19-405c-a7d9-0489ccd9dccd-kube-api-access-nkzq8\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v\" (UID: \"676e924d-1a19-405c-a7d9-0489ccd9dccd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:00.454619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.454586    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7swp\" (UniqueName: \"kubernetes.io/projected/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-kube-api-access-w7swp\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm\" (UID: \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:00.455768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.455734    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6\" (UID: \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:00.456071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.455313    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be96e79-cf81-4f39-a1e3-b987253db254-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k\" (UID: \"3be96e79-cf81-4f39-a1e3-b987253db254\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:00.457247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.457199    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5b96\" (UniqueName: \"kubernetes.io/projected/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-kube-api-access-q5b96\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn\" (UID: \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:00.462095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.462060    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xdv\" (UniqueName: \"kubernetes.io/projected/3be96e79-cf81-4f39-a1e3-b987253db254-kube-api-access-l6xdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k\" (UID: \"3be96e79-cf81-4f39-a1e3-b987253db254\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:00.463794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.463768    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nht\" (UniqueName: \"kubernetes.io/projected/e29b9634-c42c-4209-9ad5-5475064197de-kube-api-access-x5nht\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42\" (UID: \"e29b9634-c42c-4209-9ad5-5475064197de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:00.464268 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.464244    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrdf\" (UniqueName: \"kubernetes.io/projected/e03f8fd9-8386-43a9-9a6f-06e28c66e515-kube-api-access-lsrdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9\" (UID: \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:00.465641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.465614    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9cr\" (UniqueName: \"kubernetes.io/projected/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-kube-api-access-xn9cr\") pod \"maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6\" (UID: \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:00.504068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.504039    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:00.511831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.511803    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:00.548316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.548278    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a5651-9488-4a2e-bec3-c1a3d3e94069-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t\" (UID: \"493a5651-9488-4a2e-bec3-c1a3d3e94069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:00.548519 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.548408    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9s4z\" (UniqueName: \"kubernetes.io/projected/493a5651-9488-4a2e-bec3-c1a3d3e94069-kube-api-access-x9s4z\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t\" (UID: \"493a5651-9488-4a2e-bec3-c1a3d3e94069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:00.548587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.548550    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzq8\" (UniqueName: \"kubernetes.io/projected/676e924d-1a19-405c-a7d9-0489ccd9dccd-kube-api-access-nkzq8\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v\" (UID: \"676e924d-1a19-405c-a7d9-0489ccd9dccd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:00.548675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.548647    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676e924d-1a19-405c-a7d9-0489ccd9dccd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v\" (UID: \"676e924d-1a19-405c-a7d9-0489ccd9dccd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:00.549329 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.549302    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676e924d-1a19-405c-a7d9-0489ccd9dccd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v\" (UID: \"676e924d-1a19-405c-a7d9-0489ccd9dccd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:00.549836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.549812    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a5651-9488-4a2e-bec3-c1a3d3e94069-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t\" (UID: \"493a5651-9488-4a2e-bec3-c1a3d3e94069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:00.550457 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.550436    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:00.558332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.558244    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9s4z\" (UniqueName: \"kubernetes.io/projected/493a5651-9488-4a2e-bec3-c1a3d3e94069-kube-api-access-x9s4z\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t\" (UID: \"493a5651-9488-4a2e-bec3-c1a3d3e94069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:00.559569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.559543    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:00.564283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.563102    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzq8\" (UniqueName: \"kubernetes.io/projected/676e924d-1a19-405c-a7d9-0489ccd9dccd-kube-api-access-nkzq8\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v\" (UID: \"676e924d-1a19-405c-a7d9-0489ccd9dccd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:00.571969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.571903    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:00.592848 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.592563    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:00.601293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.601267    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:00.608478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.608408    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:00.996065 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:00:00.995755    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29b9634_c42c_4209_9ad5_5475064197de.slice/crio-3d2185fdc9d13ed1277b84906d1d41774ebbb1ec284a729fbbd570b80e0f497b WatchSource:0}: Error finding container 3d2185fdc9d13ed1277b84906d1d41774ebbb1ec284a729fbbd570b80e0f497b: Status 404 returned error can't find the container with id 3d2185fdc9d13ed1277b84906d1d41774ebbb1ec284a729fbbd570b80e0f497b
Nov 03 11:00:00.998323 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:00.998215    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"]
Nov 03 11:00:00.999003 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:00:00.998977    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da1a886_4b4e_465f_bcdf_2110e9a73ab8.slice/crio-c1954b2f91ad168795a7a1a52445a7d7d249bb835a8d8db48dde19653690f128 WatchSource:0}: Error finding container c1954b2f91ad168795a7a1a52445a7d7d249bb835a8d8db48dde19653690f128: Status 404 returned error can't find the container with id c1954b2f91ad168795a7a1a52445a7d7d249bb835a8d8db48dde19653690f128
Nov 03 11:00:01.004652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.004622    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"]
Nov 03 11:00:01.044567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.044542    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"]
Nov 03 11:00:01.044668 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:00:01.044578    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73a11ca_fd8c_40e6_a101_59177c7dc9f7.slice/crio-7ccc08a2d186da122c0d973a808d836eb20113f15205f7cde743747349582eee WatchSource:0}: Error finding container 7ccc08a2d186da122c0d973a808d836eb20113f15205f7cde743747349582eee: Status 404 returned error can't find the container with id 7ccc08a2d186da122c0d973a808d836eb20113f15205f7cde743747349582eee
Nov 03 11:00:01.054371 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.054344    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" event={"ID":"7da1a886-4b4e-465f-bcdf-2110e9a73ab8","Type":"ContainerStarted","Data":"c1954b2f91ad168795a7a1a52445a7d7d249bb835a8d8db48dde19653690f128"}
Nov 03 11:00:01.055597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.055571    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" event={"ID":"e29b9634-c42c-4209-9ad5-5475064197de","Type":"ContainerStarted","Data":"3d2185fdc9d13ed1277b84906d1d41774ebbb1ec284a729fbbd570b80e0f497b"}
Nov 03 11:00:01.056849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.056827    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" event={"ID":"b73a11ca-fd8c-40e6-a101-59177c7dc9f7","Type":"ContainerStarted","Data":"7ccc08a2d186da122c0d973a808d836eb20113f15205f7cde743747349582eee"}
Nov 03 11:00:01.088938 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.088889    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"]
Nov 03 11:00:01.116408 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.116375    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"]
Nov 03 11:00:01.122067 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.122044    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"]
Nov 03 11:00:01.225001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.224874    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"]
Nov 03 11:00:01.232669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:01.232543    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"]
Nov 03 11:00:02.070210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.070172    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" event={"ID":"676e924d-1a19-405c-a7d9-0489ccd9dccd","Type":"ContainerStarted","Data":"b32f8bdb2e89fb414fe50b2b788b452ddd3919846201f62179b8a438e9254480"}
Nov 03 11:00:02.070394 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.070232    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" event={"ID":"676e924d-1a19-405c-a7d9-0489ccd9dccd","Type":"ContainerStarted","Data":"69f8c93e08f1c8ee1e74e079e6d42e98707cfb4f0266fda2de50039cb2ce775a"}
Nov 03 11:00:02.072366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.072004    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" event={"ID":"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2","Type":"ContainerStarted","Data":"be4ed90331121f489975c33c67170f72424263e0772e685c412e64ff9191d023"}
Nov 03 11:00:02.072366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.072054    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" event={"ID":"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2","Type":"ContainerStarted","Data":"268333bda7e92148c3b25819d0a347953c580388cafff382ad0206c122abaab7"}
Nov 03 11:00:02.073708 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.073682    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" event={"ID":"3be96e79-cf81-4f39-a1e3-b987253db254","Type":"ContainerStarted","Data":"1313f85f8ecab18e4dc67739afda9001906f2738bc430dee88be605b8fab0b78"}
Nov 03 11:00:02.073836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.073723    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" event={"ID":"3be96e79-cf81-4f39-a1e3-b987253db254","Type":"ContainerStarted","Data":"981032f358d749f28b9a565bff2dfb50d972bc310bd4ae9c962e314f1861453c"}
Nov 03 11:00:02.075270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.075242    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" event={"ID":"7da1a886-4b4e-465f-bcdf-2110e9a73ab8","Type":"ContainerStarted","Data":"66e1a6217960ca42c10b90761858efefad2f12f2f49b1257a9bf693d61c05adc"}
Nov 03 11:00:02.077839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.077813    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" event={"ID":"e29b9634-c42c-4209-9ad5-5475064197de","Type":"ContainerStarted","Data":"9141e05f2cd9c4afefd33371348661b3f92e18923a6ce86377dbe9f75e0cb48f"}
Nov 03 11:00:02.079911 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.079883    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" event={"ID":"493a5651-9488-4a2e-bec3-c1a3d3e94069","Type":"ContainerStarted","Data":"6758bc0081d47a5baf862550925954600e6c94b615c1946f770cb7d454d45a94"}
Nov 03 11:00:02.080084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.079946    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" event={"ID":"493a5651-9488-4a2e-bec3-c1a3d3e94069","Type":"ContainerStarted","Data":"83a3077dd28fd983ac3134fed09d5831fe867fed622a6154526fdb8fcff84d82"}
Nov 03 11:00:02.082965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.082894    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" event={"ID":"e03f8fd9-8386-43a9-9a6f-06e28c66e515","Type":"ContainerStarted","Data":"6f76f0d2af548cb462f1d79169d19b71daaac192f1df518dbd9d5762cd0027f9"}
Nov 03 11:00:02.083130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.083114    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" event={"ID":"e03f8fd9-8386-43a9-9a6f-06e28c66e515","Type":"ContainerStarted","Data":"7532e4b69286572bf761806d23b8f7a6e1869c5028417d53efd968fb6125c1f5"}
Nov 03 11:00:02.091531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.091496    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" event={"ID":"b73a11ca-fd8c-40e6-a101-59177c7dc9f7","Type":"ContainerStarted","Data":"d14c6766fa21917218829b3d7aa72cc59b1670623cb8cb0a12b5171254739138"}
Nov 03 11:00:02.093236 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.093192    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" podStartSLOduration=2.09317914 podStartE2EDuration="2.09317914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.084967371 +0000 UTC m=+1845.989821812" watchObservedRunningTime="2026-11-03 11:00:02.09317914 +0000 UTC m=+1845.998033580"
Nov 03 11:00:02.101206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.101154    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" podStartSLOduration=2.101139602 podStartE2EDuration="2.101139602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.09947967 +0000 UTC m=+1846.004334113" watchObservedRunningTime="2026-11-03 11:00:02.101139602 +0000 UTC m=+1846.005994043"
Nov 03 11:00:02.116314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.116186    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" podStartSLOduration=2.116168073 podStartE2EDuration="2.116168073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.113119778 +0000 UTC m=+1846.017974240" watchObservedRunningTime="2026-11-03 11:00:02.116168073 +0000 UTC m=+1846.021022514"
Nov 03 11:00:02.128696 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.128630    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" podStartSLOduration=2.128610101 podStartE2EDuration="2.128610101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.127608125 +0000 UTC m=+1846.032462562" watchObservedRunningTime="2026-11-03 11:00:02.128610101 +0000 UTC m=+1846.033464543"
Nov 03 11:00:02.144554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.144495    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" podStartSLOduration=2.144480611 podStartE2EDuration="2.144480611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.143440411 +0000 UTC m=+1846.048294854" watchObservedRunningTime="2026-11-03 11:00:02.144480611 +0000 UTC m=+1846.049335082"
Nov 03 11:00:02.179758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.179702    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" podStartSLOduration=2.179686478 podStartE2EDuration="2.179686478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.161702862 +0000 UTC m=+1846.066557305" watchObservedRunningTime="2026-11-03 11:00:02.179686478 +0000 UTC m=+1846.084540920"
Nov 03 11:00:02.180229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.180206    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" podStartSLOduration=2.180201092 podStartE2EDuration="2.180201092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.178280513 +0000 UTC m=+1846.083134955" watchObservedRunningTime="2026-11-03 11:00:02.180201092 +0000 UTC m=+1846.085055532"
Nov 03 11:00:02.199501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:02.199447    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" podStartSLOduration=2.19943027 podStartE2EDuration="2.19943027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:02.196250559 +0000 UTC m=+1846.101105000" watchObservedRunningTime="2026-11-03 11:00:02.19943027 +0000 UTC m=+1846.104284711"
Nov 03 11:00:03.100101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.100058    5393 generic.go:356] "Generic (PLEG): container finished" podID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerID="b32f8bdb2e89fb414fe50b2b788b452ddd3919846201f62179b8a438e9254480" exitCode=60
Nov 03 11:00:03.100573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.100128    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" event={"ID":"676e924d-1a19-405c-a7d9-0489ccd9dccd","Type":"ContainerDied","Data":"b32f8bdb2e89fb414fe50b2b788b452ddd3919846201f62179b8a438e9254480"}
Nov 03 11:00:03.100573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.100451    5393 scope.go:122] "RemoveContainer" containerID="b32f8bdb2e89fb414fe50b2b788b452ddd3919846201f62179b8a438e9254480"
Nov 03 11:00:03.102066 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.102043    5393 generic.go:356] "Generic (PLEG): container finished" podID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerID="be4ed90331121f489975c33c67170f72424263e0772e685c412e64ff9191d023" exitCode=60
Nov 03 11:00:03.102198 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.102125    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" event={"ID":"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2","Type":"ContainerDied","Data":"be4ed90331121f489975c33c67170f72424263e0772e685c412e64ff9191d023"}
Nov 03 11:00:03.102425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.102409    5393 scope.go:122] "RemoveContainer" containerID="be4ed90331121f489975c33c67170f72424263e0772e685c412e64ff9191d023"
Nov 03 11:00:03.104091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.103959    5393 generic.go:356] "Generic (PLEG): container finished" podID="3be96e79-cf81-4f39-a1e3-b987253db254" containerID="1313f85f8ecab18e4dc67739afda9001906f2738bc430dee88be605b8fab0b78" exitCode=60
Nov 03 11:00:03.104091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.104032    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" event={"ID":"3be96e79-cf81-4f39-a1e3-b987253db254","Type":"ContainerDied","Data":"1313f85f8ecab18e4dc67739afda9001906f2738bc430dee88be605b8fab0b78"}
Nov 03 11:00:03.104304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.104267    5393 scope.go:122] "RemoveContainer" containerID="1313f85f8ecab18e4dc67739afda9001906f2738bc430dee88be605b8fab0b78"
Nov 03 11:00:03.105787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.105756    5393 generic.go:356] "Generic (PLEG): container finished" podID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerID="66e1a6217960ca42c10b90761858efefad2f12f2f49b1257a9bf693d61c05adc" exitCode=60
Nov 03 11:00:03.105893 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.105834    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" event={"ID":"7da1a886-4b4e-465f-bcdf-2110e9a73ab8","Type":"ContainerDied","Data":"66e1a6217960ca42c10b90761858efefad2f12f2f49b1257a9bf693d61c05adc"}
Nov 03 11:00:03.106186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.106052    5393 scope.go:122] "RemoveContainer" containerID="66e1a6217960ca42c10b90761858efefad2f12f2f49b1257a9bf693d61c05adc"
Nov 03 11:00:03.108031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.108010    5393 generic.go:356] "Generic (PLEG): container finished" podID="e29b9634-c42c-4209-9ad5-5475064197de" containerID="9141e05f2cd9c4afefd33371348661b3f92e18923a6ce86377dbe9f75e0cb48f" exitCode=60
Nov 03 11:00:03.108130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.108090    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" event={"ID":"e29b9634-c42c-4209-9ad5-5475064197de","Type":"ContainerDied","Data":"9141e05f2cd9c4afefd33371348661b3f92e18923a6ce86377dbe9f75e0cb48f"}
Nov 03 11:00:03.108332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.108313    5393 scope.go:122] "RemoveContainer" containerID="9141e05f2cd9c4afefd33371348661b3f92e18923a6ce86377dbe9f75e0cb48f"
Nov 03 11:00:03.109972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.109950    5393 generic.go:356] "Generic (PLEG): container finished" podID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerID="6758bc0081d47a5baf862550925954600e6c94b615c1946f770cb7d454d45a94" exitCode=60
Nov 03 11:00:03.110079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.110025    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" event={"ID":"493a5651-9488-4a2e-bec3-c1a3d3e94069","Type":"ContainerDied","Data":"6758bc0081d47a5baf862550925954600e6c94b615c1946f770cb7d454d45a94"}
Nov 03 11:00:03.110227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.110209    5393 scope.go:122] "RemoveContainer" containerID="6758bc0081d47a5baf862550925954600e6c94b615c1946f770cb7d454d45a94"
Nov 03 11:00:03.111864 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.111837    5393 generic.go:356] "Generic (PLEG): container finished" podID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerID="6f76f0d2af548cb462f1d79169d19b71daaac192f1df518dbd9d5762cd0027f9" exitCode=60
Nov 03 11:00:03.111971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.111948    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" event={"ID":"e03f8fd9-8386-43a9-9a6f-06e28c66e515","Type":"ContainerDied","Data":"6f76f0d2af548cb462f1d79169d19b71daaac192f1df518dbd9d5762cd0027f9"}
Nov 03 11:00:03.112169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.112152    5393 scope.go:122] "RemoveContainer" containerID="6f76f0d2af548cb462f1d79169d19b71daaac192f1df518dbd9d5762cd0027f9"
Nov 03 11:00:03.115375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.114548    5393 generic.go:356] "Generic (PLEG): container finished" podID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerID="d14c6766fa21917218829b3d7aa72cc59b1670623cb8cb0a12b5171254739138" exitCode=60
Nov 03 11:00:03.115375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.114577    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" event={"ID":"b73a11ca-fd8c-40e6-a101-59177c7dc9f7","Type":"ContainerDied","Data":"d14c6766fa21917218829b3d7aa72cc59b1670623cb8cb0a12b5171254739138"}
Nov 03 11:00:03.115375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:03.114777    5393 scope.go:122] "RemoveContainer" containerID="d14c6766fa21917218829b3d7aa72cc59b1670623cb8cb0a12b5171254739138"
Nov 03 11:00:04.031422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.031326    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"]
Nov 03 11:00:04.059310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.059253    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed0b3e82-g2sbn"]
Nov 03 11:00:04.126174 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.126127    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" event={"ID":"3be96e79-cf81-4f39-a1e3-b987253db254","Type":"ContainerStarted","Data":"824b0969c52774ed73ef89658891e5c77689b7c51a213da7d0daed48e7e47fd9"}
Nov 03 11:00:04.128034 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.128002    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" event={"ID":"7da1a886-4b4e-465f-bcdf-2110e9a73ab8","Type":"ContainerStarted","Data":"b4f6e4552fde00e438e8d5e84ca5cc51e88765941532e4b322e306d73ff6da0b"}
Nov 03 11:00:04.130110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.130085    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" event={"ID":"e29b9634-c42c-4209-9ad5-5475064197de","Type":"ContainerStarted","Data":"eebeaa58ff048e5335654b2a0941e038b65607849757bbec551d56e77ac690a5"}
Nov 03 11:00:04.132267 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.132206    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" event={"ID":"493a5651-9488-4a2e-bec3-c1a3d3e94069","Type":"ContainerStarted","Data":"07a063c56c4ed91536101e34636d8771efb625fd3c70fa1c38da69eab80f2b9e"}
Nov 03 11:00:04.134259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.134235    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" event={"ID":"e03f8fd9-8386-43a9-9a6f-06e28c66e515","Type":"ContainerStarted","Data":"f956073686ac325ed8e41da1d6b94cd33d743de8720fde56fe264297cf760d76"}
Nov 03 11:00:04.136050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.136025    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" event={"ID":"b73a11ca-fd8c-40e6-a101-59177c7dc9f7","Type":"ContainerStarted","Data":"afcbe3fc453aae83865fdf81d29256cbfbcdf7be0e0c32a98db1bdb86468f016"}
Nov 03 11:00:04.137978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.137953    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" event={"ID":"676e924d-1a19-405c-a7d9-0489ccd9dccd","Type":"ContainerStarted","Data":"049ee528ff2b940bc0029c5249812d0c7a2ebd2c754469ca5e6773caa21205d7"}
Nov 03 11:00:04.139701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.139682    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" event={"ID":"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2","Type":"ContainerStarted","Data":"2845a8ea40401fa58fc12489a3abfbd67494393b81f6bc61f40490e8f7303e39"}
Nov 03 11:00:04.232026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:04.231993    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9141e9-c8b0-4315-9197-235304be49c7" path="/var/lib/kubelet/pods/ad9141e9-c8b0-4315-9197-235304be49c7/volumes"
Nov 03 11:00:05.149126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.149090    5393 generic.go:356] "Generic (PLEG): container finished" podID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerID="afcbe3fc453aae83865fdf81d29256cbfbcdf7be0e0c32a98db1bdb86468f016" exitCode=60
Nov 03 11:00:05.149639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.149136    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" event={"ID":"b73a11ca-fd8c-40e6-a101-59177c7dc9f7","Type":"ContainerDied","Data":"afcbe3fc453aae83865fdf81d29256cbfbcdf7be0e0c32a98db1bdb86468f016"}
Nov 03 11:00:05.149639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.149224    5393 scope.go:122] "RemoveContainer" containerID="d14c6766fa21917218829b3d7aa72cc59b1670623cb8cb0a12b5171254739138"
Nov 03 11:00:05.149639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.149547    5393 scope.go:122] "RemoveContainer" containerID="afcbe3fc453aae83865fdf81d29256cbfbcdf7be0e0c32a98db1bdb86468f016"
Nov 03 11:00:05.150006 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:05.149986    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn_odh-ai-gateway-infra(b73a11ca-fd8c-40e6-a101-59177c7dc9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7"
Nov 03 11:00:05.151373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.151351    5393 generic.go:356] "Generic (PLEG): container finished" podID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerID="049ee528ff2b940bc0029c5249812d0c7a2ebd2c754469ca5e6773caa21205d7" exitCode=60
Nov 03 11:00:05.151471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.151419    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" event={"ID":"676e924d-1a19-405c-a7d9-0489ccd9dccd","Type":"ContainerDied","Data":"049ee528ff2b940bc0029c5249812d0c7a2ebd2c754469ca5e6773caa21205d7"}
Nov 03 11:00:05.151741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.151725    5393 scope.go:122] "RemoveContainer" containerID="049ee528ff2b940bc0029c5249812d0c7a2ebd2c754469ca5e6773caa21205d7"
Nov 03 11:00:05.152110 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:05.152089    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v_odh-ai-gateway-infra(676e924d-1a19-405c-a7d9-0489ccd9dccd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd"
Nov 03 11:00:05.153493 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.153472    5393 generic.go:356] "Generic (PLEG): container finished" podID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerID="2845a8ea40401fa58fc12489a3abfbd67494393b81f6bc61f40490e8f7303e39" exitCode=60
Nov 03 11:00:05.153583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.153498    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" event={"ID":"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2","Type":"ContainerDied","Data":"2845a8ea40401fa58fc12489a3abfbd67494393b81f6bc61f40490e8f7303e39"}
Nov 03 11:00:05.153865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.153846    5393 scope.go:122] "RemoveContainer" containerID="2845a8ea40401fa58fc12489a3abfbd67494393b81f6bc61f40490e8f7303e39"
Nov 03 11:00:05.154285 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:05.154232    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm_odh-ai-gateway-infra(1cbac1ba-fad2-48f6-a5e9-0c38df672ca2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2"
Nov 03 11:00:05.155773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.155750    5393 generic.go:356] "Generic (PLEG): container finished" podID="3be96e79-cf81-4f39-a1e3-b987253db254" containerID="824b0969c52774ed73ef89658891e5c77689b7c51a213da7d0daed48e7e47fd9" exitCode=60
Nov 03 11:00:05.155872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.155852    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" event={"ID":"3be96e79-cf81-4f39-a1e3-b987253db254","Type":"ContainerDied","Data":"824b0969c52774ed73ef89658891e5c77689b7c51a213da7d0daed48e7e47fd9"}
Nov 03 11:00:05.156218 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.156196    5393 scope.go:122] "RemoveContainer" containerID="824b0969c52774ed73ef89658891e5c77689b7c51a213da7d0daed48e7e47fd9"
Nov 03 11:00:05.156591 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:05.156562    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k_odh-ai-gateway-infra(3be96e79-cf81-4f39-a1e3-b987253db254)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" podUID="3be96e79-cf81-4f39-a1e3-b987253db254"
Nov 03 11:00:05.157909 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.157886    5393 generic.go:356] "Generic (PLEG): container finished" podID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerID="b4f6e4552fde00e438e8d5e84ca5cc51e88765941532e4b322e306d73ff6da0b" exitCode=60
Nov 03 11:00:05.158059 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.157975    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" event={"ID":"7da1a886-4b4e-465f-bcdf-2110e9a73ab8","Type":"ContainerDied","Data":"b4f6e4552fde00e438e8d5e84ca5cc51e88765941532e4b322e306d73ff6da0b"}
Nov 03 11:00:05.158396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.158217    5393 scope.go:122] "RemoveContainer" containerID="b4f6e4552fde00e438e8d5e84ca5cc51e88765941532e4b322e306d73ff6da0b"
Nov 03 11:00:05.158550 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:05.158526    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6_odh-ai-gateway-infra(7da1a886-4b4e-465f-bcdf-2110e9a73ab8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8"
Nov 03 11:00:05.160521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.160500    5393 generic.go:356] "Generic (PLEG): container finished" podID="e29b9634-c42c-4209-9ad5-5475064197de" containerID="eebeaa58ff048e5335654b2a0941e038b65607849757bbec551d56e77ac690a5" exitCode=60
Nov 03 11:00:05.160613 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.160524    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" event={"ID":"e29b9634-c42c-4209-9ad5-5475064197de","Type":"ContainerDied","Data":"eebeaa58ff048e5335654b2a0941e038b65607849757bbec551d56e77ac690a5"}
Nov 03 11:00:05.160876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.160835    5393 scope.go:122] "RemoveContainer" containerID="eebeaa58ff048e5335654b2a0941e038b65607849757bbec551d56e77ac690a5"
Nov 03 11:00:05.161233 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:05.161204    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42_odh-ai-gateway-infra(e29b9634-c42c-4209-9ad5-5475064197de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" podUID="e29b9634-c42c-4209-9ad5-5475064197de"
Nov 03 11:00:05.162816 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.162793    5393 generic.go:356] "Generic (PLEG): container finished" podID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerID="07a063c56c4ed91536101e34636d8771efb625fd3c70fa1c38da69eab80f2b9e" exitCode=60
Nov 03 11:00:05.162959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.162875    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" event={"ID":"493a5651-9488-4a2e-bec3-c1a3d3e94069","Type":"ContainerDied","Data":"07a063c56c4ed91536101e34636d8771efb625fd3c70fa1c38da69eab80f2b9e"}
Nov 03 11:00:05.164195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.163885    5393 scope.go:122] "RemoveContainer" containerID="07a063c56c4ed91536101e34636d8771efb625fd3c70fa1c38da69eab80f2b9e"
Nov 03 11:00:05.164438 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:05.164414    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t_odh-ai-gateway-infra(493a5651-9488-4a2e-bec3-c1a3d3e94069)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069"
Nov 03 11:00:05.167239 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.166397    5393 generic.go:356] "Generic (PLEG): container finished" podID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerID="f956073686ac325ed8e41da1d6b94cd33d743de8720fde56fe264297cf760d76" exitCode=60
Nov 03 11:00:05.167239 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.166473    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" event={"ID":"e03f8fd9-8386-43a9-9a6f-06e28c66e515","Type":"ContainerDied","Data":"f956073686ac325ed8e41da1d6b94cd33d743de8720fde56fe264297cf760d76"}
Nov 03 11:00:05.167239 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.166841    5393 scope.go:122] "RemoveContainer" containerID="f956073686ac325ed8e41da1d6b94cd33d743de8720fde56fe264297cf760d76"
Nov 03 11:00:05.167239 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:05.167185    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9_odh-ai-gateway-infra(e03f8fd9-8386-43a9-9a6f-06e28c66e515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515"
Nov 03 11:00:05.167702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.167631    5393 scope.go:122] "RemoveContainer" containerID="b32f8bdb2e89fb414fe50b2b788b452ddd3919846201f62179b8a438e9254480"
Nov 03 11:00:05.185181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.185158    5393 scope.go:122] "RemoveContainer" containerID="be4ed90331121f489975c33c67170f72424263e0772e685c412e64ff9191d023"
Nov 03 11:00:05.199326 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.199304    5393 scope.go:122] "RemoveContainer" containerID="1313f85f8ecab18e4dc67739afda9001906f2738bc430dee88be605b8fab0b78"
Nov 03 11:00:05.214004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.213979    5393 scope.go:122] "RemoveContainer" containerID="66e1a6217960ca42c10b90761858efefad2f12f2f49b1257a9bf693d61c05adc"
Nov 03 11:00:05.228983 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.228891    5393 scope.go:122] "RemoveContainer" containerID="9141e05f2cd9c4afefd33371348661b3f92e18923a6ce86377dbe9f75e0cb48f"
Nov 03 11:00:05.245672 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.245647    5393 scope.go:122] "RemoveContainer" containerID="6758bc0081d47a5baf862550925954600e6c94b615c1946f770cb7d454d45a94"
Nov 03 11:00:05.260471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:05.260446    5393 scope.go:122] "RemoveContainer" containerID="6f76f0d2af548cb462f1d79169d19b71daaac192f1df518dbd9d5762cd0027f9"
Nov 03 11:00:06.176702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:06.176677    5393 scope.go:122] "RemoveContainer" containerID="824b0969c52774ed73ef89658891e5c77689b7c51a213da7d0daed48e7e47fd9"
Nov 03 11:00:06.177171 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:06.177088    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k_odh-ai-gateway-infra(3be96e79-cf81-4f39-a1e3-b987253db254)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" podUID="3be96e79-cf81-4f39-a1e3-b987253db254"
Nov 03 11:00:06.178387 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:06.178361    5393 scope.go:122] "RemoveContainer" containerID="b4f6e4552fde00e438e8d5e84ca5cc51e88765941532e4b322e306d73ff6da0b"
Nov 03 11:00:06.178731 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:06.178710    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6_odh-ai-gateway-infra(7da1a886-4b4e-465f-bcdf-2110e9a73ab8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8"
Nov 03 11:00:06.180074 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:06.180055    5393 scope.go:122] "RemoveContainer" containerID="eebeaa58ff048e5335654b2a0941e038b65607849757bbec551d56e77ac690a5"
Nov 03 11:00:06.180366 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:06.180348    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42_odh-ai-gateway-infra(e29b9634-c42c-4209-9ad5-5475064197de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" podUID="e29b9634-c42c-4209-9ad5-5475064197de"
Nov 03 11:00:06.181641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:06.181625    5393 scope.go:122] "RemoveContainer" containerID="07a063c56c4ed91536101e34636d8771efb625fd3c70fa1c38da69eab80f2b9e"
Nov 03 11:00:06.181900 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:06.181877    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t_odh-ai-gateway-infra(493a5651-9488-4a2e-bec3-c1a3d3e94069)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069"
Nov 03 11:00:06.183273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:06.183256    5393 scope.go:122] "RemoveContainer" containerID="f956073686ac325ed8e41da1d6b94cd33d743de8720fde56fe264297cf760d76"
Nov 03 11:00:06.183505 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:06.183486    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9_odh-ai-gateway-infra(e03f8fd9-8386-43a9-9a6f-06e28c66e515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515"
Nov 03 11:00:06.184901 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:06.184881    5393 scope.go:122] "RemoveContainer" containerID="afcbe3fc453aae83865fdf81d29256cbfbcdf7be0e0c32a98db1bdb86468f016"
Nov 03 11:00:06.185156 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:06.185136    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn_odh-ai-gateway-infra(b73a11ca-fd8c-40e6-a101-59177c7dc9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7"
Nov 03 11:00:06.186621 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:06.186605    5393 scope.go:122] "RemoveContainer" containerID="049ee528ff2b940bc0029c5249812d0c7a2ebd2c754469ca5e6773caa21205d7"
Nov 03 11:00:06.186844 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:06.186825    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v_odh-ai-gateway-infra(676e924d-1a19-405c-a7d9-0489ccd9dccd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd"
Nov 03 11:00:06.188312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:06.188297    5393 scope.go:122] "RemoveContainer" containerID="2845a8ea40401fa58fc12489a3abfbd67494393b81f6bc61f40490e8f7303e39"
Nov 03 11:00:06.188492 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:06.188476    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm_odh-ai-gateway-infra(1cbac1ba-fad2-48f6-a5e9-0c38df672ca2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2"
Nov 03 11:00:13.191296 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:13.191251    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:00:15.224111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:15.224073    5393 scope.go:122] "RemoveContainer" containerID="b4f6e4552fde00e438e8d5e84ca5cc51e88765941532e4b322e306d73ff6da0b"
Nov 03 11:00:15.226980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:15.226151    5393 scope.go:122] "RemoveContainer" containerID="f956073686ac325ed8e41da1d6b94cd33d743de8720fde56fe264297cf760d76"
Nov 03 11:00:15.227061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:15.227029    5393 scope.go:122] "RemoveContainer" containerID="afcbe3fc453aae83865fdf81d29256cbfbcdf7be0e0c32a98db1bdb86468f016"
Nov 03 11:00:15.254235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:15.244587    5393 scope.go:122] "RemoveContainer" containerID="eebeaa58ff048e5335654b2a0941e038b65607849757bbec551d56e77ac690a5"
Nov 03 11:00:16.245950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:16.245474    5393 scope.go:122] "RemoveContainer" containerID="824b0969c52774ed73ef89658891e5c77689b7c51a213da7d0daed48e7e47fd9"
Nov 03 11:00:16.276996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:16.275724    5393 scope.go:122] "RemoveContainer" containerID="07a063c56c4ed91536101e34636d8771efb625fd3c70fa1c38da69eab80f2b9e"
Nov 03 11:00:16.452124 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:16.452088    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" event={"ID":"e03f8fd9-8386-43a9-9a6f-06e28c66e515","Type":"ContainerStarted","Data":"a980bddb620051a50cbc9685bfcb6ba0b6a6463f4a0e85e8ae2c95d5228572b6"}
Nov 03 11:00:16.456460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:16.456431    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" event={"ID":"b73a11ca-fd8c-40e6-a101-59177c7dc9f7","Type":"ContainerStarted","Data":"a43f4921a717768fe874324fc7c2cf55c4bd2331a9dd4715865ecb6aeabbab0e"}
Nov 03 11:00:16.462204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:16.462175    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" event={"ID":"7da1a886-4b4e-465f-bcdf-2110e9a73ab8","Type":"ContainerStarted","Data":"c81c041182947141c39ef37d50148fa2530ae8261a79f8d27d155f6007d877d0"}
Nov 03 11:00:16.467910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:16.467881    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" event={"ID":"e29b9634-c42c-4209-9ad5-5475064197de","Type":"ContainerStarted","Data":"26320b708f3af7dd8aabf2b2317b91f82983fd67258ef61442a2cf374a5a381c"}
Nov 03 11:00:17.223175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.223095    5393 scope.go:122] "RemoveContainer" containerID="049ee528ff2b940bc0029c5249812d0c7a2ebd2c754469ca5e6773caa21205d7"
Nov 03 11:00:17.223435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.223396    5393 scope.go:122] "RemoveContainer" containerID="2845a8ea40401fa58fc12489a3abfbd67494393b81f6bc61f40490e8f7303e39"
Nov 03 11:00:17.478459 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.478372    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" event={"ID":"493a5651-9488-4a2e-bec3-c1a3d3e94069","Type":"ContainerStarted","Data":"eb5bb2dbc29d23ad5bee52e5ddc4b93b2fb39f198dc34f0bae573e1c7b1b760e"}
Nov 03 11:00:17.481721 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.481686    5393 generic.go:356] "Generic (PLEG): container finished" podID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerID="a980bddb620051a50cbc9685bfcb6ba0b6a6463f4a0e85e8ae2c95d5228572b6" exitCode=60
Nov 03 11:00:17.481885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.481823    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" event={"ID":"e03f8fd9-8386-43a9-9a6f-06e28c66e515","Type":"ContainerDied","Data":"a980bddb620051a50cbc9685bfcb6ba0b6a6463f4a0e85e8ae2c95d5228572b6"}
Nov 03 11:00:17.481885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.481876    5393 scope.go:122] "RemoveContainer" containerID="f956073686ac325ed8e41da1d6b94cd33d743de8720fde56fe264297cf760d76"
Nov 03 11:00:17.482408 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.482387    5393 scope.go:122] "RemoveContainer" containerID="a980bddb620051a50cbc9685bfcb6ba0b6a6463f4a0e85e8ae2c95d5228572b6"
Nov 03 11:00:17.482815 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:17.482783    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9_odh-ai-gateway-infra(e03f8fd9-8386-43a9-9a6f-06e28c66e515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515"
Nov 03 11:00:17.483253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.483219    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"]
Nov 03 11:00:17.488794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.488589    5393 generic.go:356] "Generic (PLEG): container finished" podID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerID="a43f4921a717768fe874324fc7c2cf55c4bd2331a9dd4715865ecb6aeabbab0e" exitCode=60
Nov 03 11:00:17.488794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.488664    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" event={"ID":"b73a11ca-fd8c-40e6-a101-59177c7dc9f7","Type":"ContainerDied","Data":"a43f4921a717768fe874324fc7c2cf55c4bd2331a9dd4715865ecb6aeabbab0e"}
Nov 03 11:00:17.489130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.489109    5393 scope.go:122] "RemoveContainer" containerID="a43f4921a717768fe874324fc7c2cf55c4bd2331a9dd4715865ecb6aeabbab0e"
Nov 03 11:00:17.489436 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:17.489413    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn_odh-ai-gateway-infra(b73a11ca-fd8c-40e6-a101-59177c7dc9f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7"
Nov 03 11:00:17.498072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.497433    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" event={"ID":"3be96e79-cf81-4f39-a1e3-b987253db254","Type":"ContainerStarted","Data":"4635f312fe2cacd50a91b50762df400d591adaf6d9b9a1cbc6835125a9fb3e97"}
Nov 03 11:00:17.499556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.499531    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"]
Nov 03 11:00:17.501346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.501320    5393 generic.go:356] "Generic (PLEG): container finished" podID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerID="c81c041182947141c39ef37d50148fa2530ae8261a79f8d27d155f6007d877d0" exitCode=60
Nov 03 11:00:17.501474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.501397    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" event={"ID":"7da1a886-4b4e-465f-bcdf-2110e9a73ab8","Type":"ContainerDied","Data":"c81c041182947141c39ef37d50148fa2530ae8261a79f8d27d155f6007d877d0"}
Nov 03 11:00:17.501802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.501784    5393 scope.go:122] "RemoveContainer" containerID="c81c041182947141c39ef37d50148fa2530ae8261a79f8d27d155f6007d877d0"
Nov 03 11:00:17.503067 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:17.503042    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6_odh-ai-gateway-infra(7da1a886-4b4e-465f-bcdf-2110e9a73ab8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8"
Nov 03 11:00:17.504297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.504274    5393 generic.go:356] "Generic (PLEG): container finished" podID="e29b9634-c42c-4209-9ad5-5475064197de" containerID="26320b708f3af7dd8aabf2b2317b91f82983fd67258ef61442a2cf374a5a381c" exitCode=60
Nov 03 11:00:17.504369 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.504329    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" event={"ID":"e29b9634-c42c-4209-9ad5-5475064197de","Type":"ContainerDied","Data":"26320b708f3af7dd8aabf2b2317b91f82983fd67258ef61442a2cf374a5a381c"}
Nov 03 11:00:17.504705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.504687    5393 scope.go:122] "RemoveContainer" containerID="26320b708f3af7dd8aabf2b2317b91f82983fd67258ef61442a2cf374a5a381c"
Nov 03 11:00:17.505049 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:17.505028    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42_odh-ai-gateway-infra(e29b9634-c42c-4209-9ad5-5475064197de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" podUID="e29b9634-c42c-4209-9ad5-5475064197de"
Nov 03 11:00:17.532476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.532316    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"]
Nov 03 11:00:17.546813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.546777    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"]
Nov 03 11:00:17.723616 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.723590    5393 scope.go:122] "RemoveContainer" containerID="afcbe3fc453aae83865fdf81d29256cbfbcdf7be0e0c32a98db1bdb86468f016"
Nov 03 11:00:17.738879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.738852    5393 scope.go:122] "RemoveContainer" containerID="b4f6e4552fde00e438e8d5e84ca5cc51e88765941532e4b322e306d73ff6da0b"
Nov 03 11:00:17.753066 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:17.753041    5393 scope.go:122] "RemoveContainer" containerID="eebeaa58ff048e5335654b2a0941e038b65607849757bbec551d56e77ac690a5"
Nov 03 11:00:18.193211 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.193183    5393 scope.go:122] "RemoveContainer" containerID="0f1d6ba6bc3b7b46ee8b0331879fbefaa257ea27055bb41c82bf31215bdfdc85"
Nov 03 11:00:18.214869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.214847    5393 scope.go:122] "RemoveContainer" containerID="e587d7057d1060290d0ffd3e5f46de8d009ddbff4fc0a55808e5a7a7b02a7203"
Nov 03 11:00:18.506312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.506234    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"]
Nov 03 11:00:18.514109 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.514079    5393 generic.go:356] "Generic (PLEG): container finished" podID="3be96e79-cf81-4f39-a1e3-b987253db254" containerID="4635f312fe2cacd50a91b50762df400d591adaf6d9b9a1cbc6835125a9fb3e97" exitCode=60
Nov 03 11:00:18.514248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.514155    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" event={"ID":"3be96e79-cf81-4f39-a1e3-b987253db254","Type":"ContainerDied","Data":"4635f312fe2cacd50a91b50762df400d591adaf6d9b9a1cbc6835125a9fb3e97"}
Nov 03 11:00:18.514248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.514221    5393 scope.go:122] "RemoveContainer" containerID="824b0969c52774ed73ef89658891e5c77689b7c51a213da7d0daed48e7e47fd9"
Nov 03 11:00:18.514687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.514668    5393 scope.go:122] "RemoveContainer" containerID="4635f312fe2cacd50a91b50762df400d591adaf6d9b9a1cbc6835125a9fb3e97"
Nov 03 11:00:18.515145 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:18.515117    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k_odh-ai-gateway-infra(3be96e79-cf81-4f39-a1e3-b987253db254)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" podUID="3be96e79-cf81-4f39-a1e3-b987253db254"
Nov 03 11:00:18.521078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.521053    5393 generic.go:356] "Generic (PLEG): container finished" podID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerID="eb5bb2dbc29d23ad5bee52e5ddc4b93b2fb39f198dc34f0bae573e1c7b1b760e" exitCode=60
Nov 03 11:00:18.521209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.521144    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" event={"ID":"493a5651-9488-4a2e-bec3-c1a3d3e94069","Type":"ContainerDied","Data":"eb5bb2dbc29d23ad5bee52e5ddc4b93b2fb39f198dc34f0bae573e1c7b1b760e"}
Nov 03 11:00:18.530422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.530066    5393 generic.go:356] "Generic (PLEG): container finished" podID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerID="a30e94f0e8768f3f82f787b0245cbaa76a1cae060df2a5d868268e0f034a077d" exitCode=60
Nov 03 11:00:18.530531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.530506    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" event={"ID":"676e924d-1a19-405c-a7d9-0489ccd9dccd","Type":"ContainerDied","Data":"a30e94f0e8768f3f82f787b0245cbaa76a1cae060df2a5d868268e0f034a077d"}
Nov 03 11:00:18.531195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.531171    5393 scope.go:122] "RemoveContainer" containerID="a30e94f0e8768f3f82f787b0245cbaa76a1cae060df2a5d868268e0f034a077d"
Nov 03 11:00:18.531649 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:18.531620    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v_odh-ai-gateway-infra(676e924d-1a19-405c-a7d9-0489ccd9dccd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd"
Nov 03 11:00:18.538161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.538133    5393 generic.go:356] "Generic (PLEG): container finished" podID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerID="65d00d8c6330bc50f30be8dd08b1e4d83b8fbce2797983952300eba9026c51cb" exitCode=60
Nov 03 11:00:18.538332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.538190    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" event={"ID":"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2","Type":"ContainerDied","Data":"65d00d8c6330bc50f30be8dd08b1e4d83b8fbce2797983952300eba9026c51cb"}
Nov 03 11:00:18.538649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.538630    5393 scope.go:122] "RemoveContainer" containerID="65d00d8c6330bc50f30be8dd08b1e4d83b8fbce2797983952300eba9026c51cb"
Nov 03 11:00:18.539092 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:18.539064    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm_odh-ai-gateway-infra(1cbac1ba-fad2-48f6-a5e9-0c38df672ca2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2"
Nov 03 11:00:18.548140 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.545628    5393 scope.go:122] "RemoveContainer" containerID="07a063c56c4ed91536101e34636d8771efb625fd3c70fa1c38da69eab80f2b9e"
Nov 03 11:00:18.616193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.611165    5393 scope.go:122] "RemoveContainer" containerID="049ee528ff2b940bc0029c5249812d0c7a2ebd2c754469ca5e6773caa21205d7"
Nov 03 11:00:18.690949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.688154    5393 scope.go:122] "RemoveContainer" containerID="2845a8ea40401fa58fc12489a3abfbd67494393b81f6bc61f40490e8f7303e39"
Nov 03 11:00:18.749421 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.747299    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"]
Nov 03 11:00:18.855451 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.854279    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:18.959703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.959674    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29b9634-c42c-4209-9ad5-5475064197de-kube-api-access-x5nht\" (UniqueName: \"kubernetes.io/projected/e29b9634-c42c-4209-9ad5-5475064197de-kube-api-access-x5nht\") pod \"e29b9634-c42c-4209-9ad5-5475064197de\" (UID: \"e29b9634-c42c-4209-9ad5-5475064197de\") "
Nov 03 11:00:18.960024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.960001    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29b9634-c42c-4209-9ad5-5475064197de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29b9634-c42c-4209-9ad5-5475064197de-maas-api-service-ca\") pod \"e29b9634-c42c-4209-9ad5-5475064197de\" (UID: \"e29b9634-c42c-4209-9ad5-5475064197de\") "
Nov 03 11:00:18.961461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.961438    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29b9634-c42c-4209-9ad5-5475064197de-maas-api-service-ca" pod "e29b9634-c42c-4209-9ad5-5475064197de" (UID: "e29b9634-c42c-4209-9ad5-5475064197de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:18.964192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.964172    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29b9634-c42c-4209-9ad5-5475064197de-kube-api-access-x5nht" pod "e29b9634-c42c-4209-9ad5-5475064197de" (UID: "e29b9634-c42c-4209-9ad5-5475064197de"). InnerVolumeSpecName "kube-api-access-x5nht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:18.985130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.985099    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:18.994632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:18.994603    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:19.006594 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.006563    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:19.016743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.016711    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:19.017836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.017812    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"]
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018856    5393 cpu_manager.go:414] "containerMap: removing container" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018880    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018898    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018906    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018937    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018945    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018966    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018972    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018984    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.018991    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019004    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019011    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019051    5393 cpu_manager.go:414] "containerMap: removing container" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019058    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019069    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019075    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019091    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019098    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019113    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019119    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019144    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:19.019273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019151    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019305    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019317    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019332    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019341    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019348    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019359    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019367    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019376    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019385    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019395    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:19.020701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.019411    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:19.022895 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.022874    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.027820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.027364    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:00:19.027995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.027977    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"]
Nov 03 11:00:19.041627 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.041606    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"]
Nov 03 11:00:19.053184 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.053122    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e615c94-gq2gm"]
Nov 03 11:00:19.063179 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.063146    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03f8fd9-8386-43a9-9a6f-06e28c66e515-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f8fd9-8386-43a9-9a6f-06e28c66e515-maas-api-service-ca\") pod \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\" (UID: \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\") "
Nov 03 11:00:19.063288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.063222    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493a5651-9488-4a2e-bec3-c1a3d3e94069-kube-api-access-x9s4z\" (UniqueName: \"kubernetes.io/projected/493a5651-9488-4a2e-bec3-c1a3d3e94069-kube-api-access-x9s4z\") pod \"493a5651-9488-4a2e-bec3-c1a3d3e94069\" (UID: \"493a5651-9488-4a2e-bec3-c1a3d3e94069\") "
Nov 03 11:00:19.063334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.063313    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03f8fd9-8386-43a9-9a6f-06e28c66e515-kube-api-access-lsrdf\" (UniqueName: \"kubernetes.io/projected/e03f8fd9-8386-43a9-9a6f-06e28c66e515-kube-api-access-lsrdf\") pod \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\" (UID: \"e03f8fd9-8386-43a9-9a6f-06e28c66e515\") "
Nov 03 11:00:19.063427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.063411    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493a5651-9488-4a2e-bec3-c1a3d3e94069-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a5651-9488-4a2e-bec3-c1a3d3e94069-maas-api-service-ca\") pod \"493a5651-9488-4a2e-bec3-c1a3d3e94069\" (UID: \"493a5651-9488-4a2e-bec3-c1a3d3e94069\") "
Nov 03 11:00:19.063536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.063513    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03f8fd9-8386-43a9-9a6f-06e28c66e515-maas-api-service-ca" pod "e03f8fd9-8386-43a9-9a6f-06e28c66e515" (UID: "e03f8fd9-8386-43a9-9a6f-06e28c66e515"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:19.064133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.064110    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5nht\" (UniqueName: \"kubernetes.io/projected/e29b9634-c42c-4209-9ad5-5475064197de-kube-api-access-x5nht\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.064305 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.064289    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f8fd9-8386-43a9-9a6f-06e28c66e515-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.064409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.064397    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29b9634-c42c-4209-9ad5-5475064197de-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.064507 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.064486    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493a5651-9488-4a2e-bec3-c1a3d3e94069-maas-api-service-ca" pod "493a5651-9488-4a2e-bec3-c1a3d3e94069" (UID: "493a5651-9488-4a2e-bec3-c1a3d3e94069"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:19.066500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.066471    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493a5651-9488-4a2e-bec3-c1a3d3e94069-kube-api-access-x9s4z" pod "493a5651-9488-4a2e-bec3-c1a3d3e94069" (UID: "493a5651-9488-4a2e-bec3-c1a3d3e94069"). InnerVolumeSpecName "kube-api-access-x9s4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:19.067518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.067493    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03f8fd9-8386-43a9-9a6f-06e28c66e515-kube-api-access-lsrdf" pod "e03f8fd9-8386-43a9-9a6f-06e28c66e515" (UID: "e03f8fd9-8386-43a9-9a6f-06e28c66e515"). InnerVolumeSpecName "kube-api-access-lsrdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:19.165613 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.165582    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-kube-api-access-xn9cr\" (UniqueName: \"kubernetes.io/projected/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-kube-api-access-xn9cr\") pod \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\" (UID: \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\") "
Nov 03 11:00:19.165800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.165620    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-maas-api-service-ca\") pod \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\" (UID: \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\") "
Nov 03 11:00:19.165911 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.165891    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-kube-api-access-q5b96\" (UniqueName: \"kubernetes.io/projected/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-kube-api-access-q5b96\") pod \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\" (UID: \"b73a11ca-fd8c-40e6-a101-59177c7dc9f7\") "
Nov 03 11:00:19.166013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.165996    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-maas-api-service-ca\") pod \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\" (UID: \"7da1a886-4b4e-465f-bcdf-2110e9a73ab8\") "
Nov 03 11:00:19.166125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166092    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-maas-api-service-ca" pod "b73a11ca-fd8c-40e6-a101-59177c7dc9f7" (UID: "b73a11ca-fd8c-40e6-a101-59177c7dc9f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:19.166390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166363    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-maas-api-service-ca" pod "7da1a886-4b4e-465f-bcdf-2110e9a73ab8" (UID: "7da1a886-4b4e-465f-bcdf-2110e9a73ab8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:19.166390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166370    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.166509 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166482    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4tx\" (UniqueName: \"kubernetes.io/projected/08fff656-ef4e-4298-9753-85b6b8f31fe2-kube-api-access-lg4tx\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.166697 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166673    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.166955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166934    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/493a5651-9488-4a2e-bec3-c1a3d3e94069-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.167000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166956    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.167000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166972    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9s4z\" (UniqueName: \"kubernetes.io/projected/493a5651-9488-4a2e-bec3-c1a3d3e94069-kube-api-access-x9s4z\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.167000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.166987    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.167108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.167012    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsrdf\" (UniqueName: \"kubernetes.io/projected/e03f8fd9-8386-43a9-9a6f-06e28c66e515-kube-api-access-lsrdf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.168210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.168186    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-kube-api-access-q5b96" pod "b73a11ca-fd8c-40e6-a101-59177c7dc9f7" (UID: "b73a11ca-fd8c-40e6-a101-59177c7dc9f7"). InnerVolumeSpecName "kube-api-access-q5b96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:19.168499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.168475    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-kube-api-access-xn9cr" pod "7da1a886-4b4e-465f-bcdf-2110e9a73ab8" (UID: "7da1a886-4b4e-465f-bcdf-2110e9a73ab8"). InnerVolumeSpecName "kube-api-access-xn9cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:19.268992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.268953    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.269187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.269008    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4tx\" (UniqueName: \"kubernetes.io/projected/08fff656-ef4e-4298-9753-85b6b8f31fe2-kube-api-access-lg4tx\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.269187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.269108    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.269283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.269245    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn9cr\" (UniqueName: \"kubernetes.io/projected/7da1a886-4b4e-465f-bcdf-2110e9a73ab8-kube-api-access-xn9cr\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.269283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.269264    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5b96\" (UniqueName: \"kubernetes.io/projected/b73a11ca-fd8c-40e6-a101-59177c7dc9f7-kube-api-access-q5b96\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:19.269647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.269627    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.271614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.271590    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.276785 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.276765    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4tx\" (UniqueName: \"kubernetes.io/projected/08fff656-ef4e-4298-9753-85b6b8f31fe2-kube-api-access-lg4tx\") pod \"maas-api-revoke-sub-035405d5-cm67h\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.367994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.360243    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:19.565971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.565525    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6" event={"ID":"7da1a886-4b4e-465f-bcdf-2110e9a73ab8","Type":"ContainerDied","Data":"c1954b2f91ad168795a7a1a52445a7d7d249bb835a8d8db48dde19653690f128"}
Nov 03 11:00:19.565971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.565628    5393 scope.go:122] "RemoveContainer" containerID="c81c041182947141c39ef37d50148fa2530ae8261a79f8d27d155f6007d877d0"
Nov 03 11:00:19.565971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.565779    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"
Nov 03 11:00:19.589500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.589160    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"
Nov 03 11:00:19.589889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.589857    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42" event={"ID":"e29b9634-c42c-4209-9ad5-5475064197de","Type":"ContainerDied","Data":"3d2185fdc9d13ed1277b84906d1d41774ebbb1ec284a729fbbd570b80e0f497b"}
Nov 03 11:00:19.596966 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.595819    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t" event={"ID":"493a5651-9488-4a2e-bec3-c1a3d3e94069","Type":"ContainerDied","Data":"83a3077dd28fd983ac3134fed09d5831fe867fed622a6154526fdb8fcff84d82"}
Nov 03 11:00:19.596966 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.595966    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"
Nov 03 11:00:19.611105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.611047    5393 scope.go:122] "RemoveContainer" containerID="26320b708f3af7dd8aabf2b2317b91f82983fd67258ef61442a2cf374a5a381c"
Nov 03 11:00:19.615127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.615054    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9" event={"ID":"e03f8fd9-8386-43a9-9a6f-06e28c66e515","Type":"ContainerDied","Data":"7532e4b69286572bf761806d23b8f7a6e1869c5028417d53efd968fb6125c1f5"}
Nov 03 11:00:19.615315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.615298    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"
Nov 03 11:00:19.621704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.621627    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn" event={"ID":"b73a11ca-fd8c-40e6-a101-59177c7dc9f7","Type":"ContainerDied","Data":"7ccc08a2d186da122c0d973a808d836eb20113f15205f7cde743747349582eee"}
Nov 03 11:00:19.621812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.621746    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"
Nov 03 11:00:19.642017 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.641988    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"]
Nov 03 11:00:19.645180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.645158    5393 scope.go:122] "RemoveContainer" containerID="eb5bb2dbc29d23ad5bee52e5ddc4b93b2fb39f198dc34f0bae573e1c7b1b760e"
Nov 03 11:00:19.648998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.648976    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e78c33-29895060-hf6d6"]
Nov 03 11:00:19.671297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.671266    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"]
Nov 03 11:00:19.676486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.676447    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895060-7rd2t"]
Nov 03 11:00:19.677088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.677070    5393 scope.go:122] "RemoveContainer" containerID="a980bddb620051a50cbc9685bfcb6ba0b6a6463f4a0e85e8ae2c95d5228572b6"
Nov 03 11:00:19.685246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.685218    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"]
Nov 03 11:00:19.879228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:19.879115    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"]
Nov 03 11:00:20.229061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.228979    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895060-2zn42"]
Nov 03 11:00:20.252155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.252121    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" path="/var/lib/kubelet/pods/493a5651-9488-4a2e-bec3-c1a3d3e94069/volumes"
Nov 03 11:00:20.252904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.252833    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a29ff0a-28dd-44d2-8cfb-18c767be8008" path="/var/lib/kubelet/pods/5a29ff0a-28dd-44d2-8cfb-18c767be8008/volumes"
Nov 03 11:00:20.253434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.253410    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" path="/var/lib/kubelet/pods/7da1a886-4b4e-465f-bcdf-2110e9a73ab8/volumes"
Nov 03 11:00:20.254083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.254060    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29b9634-c42c-4209-9ad5-5475064197de" path="/var/lib/kubelet/pods/e29b9634-c42c-4209-9ad5-5475064197de/volumes"
Nov 03 11:00:20.254454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.254437    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"]
Nov 03 11:00:20.260818 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.260795    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"]
Nov 03 11:00:20.268163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.268110    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-6012b5-29895060-69tl9"]
Nov 03 11:00:20.300227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.300191    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"]
Nov 03 11:00:20.308346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.308260    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"]
Nov 03 11:00:20.315781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.315758    5393 scope.go:122] "RemoveContainer" containerID="a43f4921a717768fe874324fc7c2cf55c4bd2331a9dd4715865ecb6aeabbab0e"
Nov 03 11:00:20.317467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.317452    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895060-t5gkn"]
Nov 03 11:00:20.331213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.331188    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:20.434698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.434657    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:20.446626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.446597    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be96e79-cf81-4f39-a1e3-b987253db254-kube-api-access-l6xdv\" (UniqueName: \"kubernetes.io/projected/3be96e79-cf81-4f39-a1e3-b987253db254-kube-api-access-l6xdv\") pod \"3be96e79-cf81-4f39-a1e3-b987253db254\" (UID: \"3be96e79-cf81-4f39-a1e3-b987253db254\") "
Nov 03 11:00:20.446801 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.446792    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be96e79-cf81-4f39-a1e3-b987253db254-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be96e79-cf81-4f39-a1e3-b987253db254-maas-api-service-ca\") pod \"3be96e79-cf81-4f39-a1e3-b987253db254\" (UID: \"3be96e79-cf81-4f39-a1e3-b987253db254\") "
Nov 03 11:00:20.447205 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.447182    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be96e79-cf81-4f39-a1e3-b987253db254-maas-api-service-ca" pod "3be96e79-cf81-4f39-a1e3-b987253db254" (UID: "3be96e79-cf81-4f39-a1e3-b987253db254"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:20.447437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.447421    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be96e79-cf81-4f39-a1e3-b987253db254-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:20.449724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.449695    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be96e79-cf81-4f39-a1e3-b987253db254-kube-api-access-l6xdv" pod "3be96e79-cf81-4f39-a1e3-b987253db254" (UID: "3be96e79-cf81-4f39-a1e3-b987253db254"). InnerVolumeSpecName "kube-api-access-l6xdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:20.491726 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.491688    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:20.548075 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.548036    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676e924d-1a19-405c-a7d9-0489ccd9dccd-kube-api-access-nkzq8\" (UniqueName: \"kubernetes.io/projected/676e924d-1a19-405c-a7d9-0489ccd9dccd-kube-api-access-nkzq8\") pod \"676e924d-1a19-405c-a7d9-0489ccd9dccd\" (UID: \"676e924d-1a19-405c-a7d9-0489ccd9dccd\") "
Nov 03 11:00:20.548292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.548272    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676e924d-1a19-405c-a7d9-0489ccd9dccd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676e924d-1a19-405c-a7d9-0489ccd9dccd-maas-api-service-ca\") pod \"676e924d-1a19-405c-a7d9-0489ccd9dccd\" (UID: \"676e924d-1a19-405c-a7d9-0489ccd9dccd\") "
Nov 03 11:00:20.548667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.548632    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676e924d-1a19-405c-a7d9-0489ccd9dccd-maas-api-service-ca" pod "676e924d-1a19-405c-a7d9-0489ccd9dccd" (UID: "676e924d-1a19-405c-a7d9-0489ccd9dccd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:20.549032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.549015    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676e924d-1a19-405c-a7d9-0489ccd9dccd-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:20.549204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.549033    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6xdv\" (UniqueName: \"kubernetes.io/projected/3be96e79-cf81-4f39-a1e3-b987253db254-kube-api-access-l6xdv\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:20.550553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.550524    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676e924d-1a19-405c-a7d9-0489ccd9dccd-kube-api-access-nkzq8" pod "676e924d-1a19-405c-a7d9-0489ccd9dccd" (UID: "676e924d-1a19-405c-a7d9-0489ccd9dccd"). InnerVolumeSpecName "kube-api-access-nkzq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:20.653299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.649895    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-maas-api-service-ca\") pod \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\" (UID: \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\") "
Nov 03 11:00:20.680306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.677620    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-kube-api-access-w7swp\" (UniqueName: \"kubernetes.io/projected/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-kube-api-access-w7swp\") pod \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\" (UID: \"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2\") "
Nov 03 11:00:20.682872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.667234    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-maas-api-service-ca" pod "1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" (UID: "1cbac1ba-fad2-48f6-a5e9-0c38df672ca2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:20.683514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.683481    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkzq8\" (UniqueName: \"kubernetes.io/projected/676e924d-1a19-405c-a7d9-0489ccd9dccd-kube-api-access-nkzq8\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:20.683637 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.683519    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:20.688210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.686643    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-kube-api-access-w7swp" pod "1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" (UID: "1cbac1ba-fad2-48f6-a5e9-0c38df672ca2"). InnerVolumeSpecName "kube-api-access-w7swp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:20.762257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.762166    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v" event={"ID":"676e924d-1a19-405c-a7d9-0489ccd9dccd","Type":"ContainerDied","Data":"69f8c93e08f1c8ee1e74e079e6d42e98707cfb4f0266fda2de50039cb2ce775a"}
Nov 03 11:00:20.762525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.762183    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"
Nov 03 11:00:20.762662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.762502    5393 scope.go:122] "RemoveContainer" containerID="a30e94f0e8768f3f82f787b0245cbaa76a1cae060df2a5d868268e0f034a077d"
Nov 03 11:00:20.763875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.763784    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm" event={"ID":"1cbac1ba-fad2-48f6-a5e9-0c38df672ca2","Type":"ContainerDied","Data":"268333bda7e92148c3b25819d0a347953c580388cafff382ad0206c122abaab7"}
Nov 03 11:00:20.764007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.763887    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"
Nov 03 11:00:20.767997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.767683    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k" event={"ID":"3be96e79-cf81-4f39-a1e3-b987253db254","Type":"ContainerDied","Data":"981032f358d749f28b9a565bff2dfb50d972bc310bd4ae9c962e314f1861453c"}
Nov 03 11:00:20.767997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.767747    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"
Nov 03 11:00:20.774703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.774680    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h" event={"ID":"08fff656-ef4e-4298-9753-85b6b8f31fe2","Type":"ContainerStarted","Data":"a56a7ac793c354324f27668822261465e2278ee35f00774918f06554d8a5ca71"}
Nov 03 11:00:20.778022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.778002    5393 scope.go:122] "RemoveContainer" containerID="65d00d8c6330bc50f30be8dd08b1e4d83b8fbce2797983952300eba9026c51cb"
Nov 03 11:00:20.784820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.784791    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7swp\" (UniqueName: \"kubernetes.io/projected/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2-kube-api-access-w7swp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:20.788966 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.788879    5393 scope.go:122] "RemoveContainer" containerID="4635f312fe2cacd50a91b50762df400d591adaf6d9b9a1cbc6835125a9fb3e97"
Nov 03 11:00:20.791794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.791764    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"]
Nov 03 11:00:20.797175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.797145    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895060-mvs4v"]
Nov 03 11:00:20.808903 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.808877    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"]
Nov 03 11:00:20.814329 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.814303    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895060-59hgm"]
Nov 03 11:00:20.828746 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.828698    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"]
Nov 03 11:00:20.832509 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:20.832481    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-e11de4-29895060-l6d2k"]
Nov 03 11:00:21.793309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:21.793205    5393 generic.go:356] "Generic (PLEG): container finished" podID="08fff656-ef4e-4298-9753-85b6b8f31fe2" containerID="7ad3acbe10f7f35bbcbfe08ca9976ac4cbf2685eb356322cda568771240351e2" exitCode=0
Nov 03 11:00:21.793309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:21.793262    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h" event={"ID":"08fff656-ef4e-4298-9753-85b6b8f31fe2","Type":"ContainerDied","Data":"7ad3acbe10f7f35bbcbfe08ca9976ac4cbf2685eb356322cda568771240351e2"}
Nov 03 11:00:22.231618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:22.231572    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" path="/var/lib/kubelet/pods/1cbac1ba-fad2-48f6-a5e9-0c38df672ca2/volumes"
Nov 03 11:00:22.232233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:22.232215    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" path="/var/lib/kubelet/pods/3be96e79-cf81-4f39-a1e3-b987253db254/volumes"
Nov 03 11:00:22.232619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:22.232605    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" path="/var/lib/kubelet/pods/676e924d-1a19-405c-a7d9-0489ccd9dccd/volumes"
Nov 03 11:00:22.233306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:22.233292    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" path="/var/lib/kubelet/pods/b73a11ca-fd8c-40e6-a101-59177c7dc9f7/volumes"
Nov 03 11:00:22.233680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:22.233665    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" path="/var/lib/kubelet/pods/e03f8fd9-8386-43a9-9a6f-06e28c66e515/volumes"
Nov 03 11:00:22.967472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:22.967440    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:23.027125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.027090    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-openshift-ca\") pod \"08fff656-ef4e-4298-9753-85b6b8f31fe2\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") "
Nov 03 11:00:23.027282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.027257    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-xks-ca\") pod \"08fff656-ef4e-4298-9753-85b6b8f31fe2\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") "
Nov 03 11:00:23.027343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.027329    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fff656-ef4e-4298-9753-85b6b8f31fe2-kube-api-access-lg4tx\" (UniqueName: \"kubernetes.io/projected/08fff656-ef4e-4298-9753-85b6b8f31fe2-kube-api-access-lg4tx\") pod \"08fff656-ef4e-4298-9753-85b6b8f31fe2\" (UID: \"08fff656-ef4e-4298-9753-85b6b8f31fe2\") "
Nov 03 11:00:23.027518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.027490    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-openshift-ca" pod "08fff656-ef4e-4298-9753-85b6b8f31fe2" (UID: "08fff656-ef4e-4298-9753-85b6b8f31fe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:23.028097 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.028024    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:23.029549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.029524    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-xks-ca" pod "08fff656-ef4e-4298-9753-85b6b8f31fe2" (UID: "08fff656-ef4e-4298-9753-85b6b8f31fe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:00:23.029661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.029640    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fff656-ef4e-4298-9753-85b6b8f31fe2-kube-api-access-lg4tx" pod "08fff656-ef4e-4298-9753-85b6b8f31fe2" (UID: "08fff656-ef4e-4298-9753-85b6b8f31fe2"). InnerVolumeSpecName "kube-api-access-lg4tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:23.137444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.136994    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fff656-ef4e-4298-9753-85b6b8f31fe2-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:23.137444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.137020    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4tx\" (UniqueName: \"kubernetes.io/projected/08fff656-ef4e-4298-9753-85b6b8f31fe2-kube-api-access-lg4tx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:23.809635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.809593    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h" event={"ID":"08fff656-ef4e-4298-9753-85b6b8f31fe2","Type":"ContainerDied","Data":"a56a7ac793c354324f27668822261465e2278ee35f00774918f06554d8a5ca71"}
Nov 03 11:00:23.809852 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.809654    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a56a7ac793c354324f27668822261465e2278ee35f00774918f06554d8a5ca71"
Nov 03 11:00:23.809852 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:23.809616    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"
Nov 03 11:00:24.905347 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.905308    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"]
Nov 03 11:00:24.906275 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906254    5393 cpu_manager.go:414] "containerMap: removing container" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:24.906275 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906275    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906287    5393 cpu_manager.go:414] "containerMap: removing container" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906292    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906299    5393 cpu_manager.go:414] "containerMap: removing container" podUID="08fff656-ef4e-4298-9753-85b6b8f31fe2" containerName="revoke-keys"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906306    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fff656-ef4e-4298-9753-85b6b8f31fe2" containerName="revoke-keys"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906322    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906330    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906354    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906359    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906369    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906374    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906400    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906405    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906414    5393 cpu_manager.go:414] "containerMap: removing container" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906419    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906427    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906431    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906443    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906447    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906455    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:24.906466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906459    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da1a886-4b4e-465f-bcdf-2110e9a73ab8" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906551    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906558    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906563    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fff656-ef4e-4298-9753-85b6b8f31fe2" containerName="revoke-keys"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906569    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906579    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906585    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906592    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73a11ca-fd8c-40e6-a101-59177c7dc9f7" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906599    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906605    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906611    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03f8fd9-8386-43a9-9a6f-06e28c66e515" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906618    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a5651-9488-4a2e-bec3-c1a3d3e94069" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906624    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:24.907430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.906629    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29b9634-c42c-4209-9ad5-5475064197de" containerName="cleanup"
Nov 03 11:00:24.910749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.910723    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:24.921127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.921102    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-185908-dockercfg-knfhz\""
Nov 03 11:00:24.922278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.921272    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"]
Nov 03 11:00:24.980008 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.979976    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgkp\" (UniqueName: \"kubernetes.io/projected/1441f96d-52b2-420c-bb88-988f13c65e65-kube-api-access-chgkp\") pod \"payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns\" (UID: \"1441f96d-52b2-420c-bb88-988f13c65e65\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:24.980349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:24.980320    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1441f96d-52b2-420c-bb88-988f13c65e65-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns\" (UID: \"1441f96d-52b2-420c-bb88-988f13c65e65\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:25.018953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.004436    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-185908\""
Nov 03 11:00:25.100941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.088908    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgkp\" (UniqueName: \"kubernetes.io/projected/1441f96d-52b2-420c-bb88-988f13c65e65-kube-api-access-chgkp\") pod \"payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns\" (UID: \"1441f96d-52b2-420c-bb88-988f13c65e65\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:25.100941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.089285    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1441f96d-52b2-420c-bb88-988f13c65e65-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns\" (UID: \"1441f96d-52b2-420c-bb88-988f13c65e65\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:25.100941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.090215    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1441f96d-52b2-420c-bb88-988f13c65e65-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns\" (UID: \"1441f96d-52b2-420c-bb88-988f13c65e65\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:25.142943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.135864    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgkp\" (UniqueName: \"kubernetes.io/projected/1441f96d-52b2-420c-bb88-988f13c65e65-kube-api-access-chgkp\") pod \"payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns\" (UID: \"1441f96d-52b2-420c-bb88-988f13c65e65\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:25.235996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.232381    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:25.586315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.586284    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"]
Nov 03 11:00:25.586640 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:00:25.586611    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1441f96d_52b2_420c_bb88_988f13c65e65.slice/crio-396ad3359d0e9f19342296f7eda678125bc8b7e52c7f21d96eb253243eae0d5f WatchSource:0}: Error finding container 396ad3359d0e9f19342296f7eda678125bc8b7e52c7f21d96eb253243eae0d5f: Status 404 returned error can't find the container with id 396ad3359d0e9f19342296f7eda678125bc8b7e52c7f21d96eb253243eae0d5f
Nov 03 11:00:25.870282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.870242    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns" event={"ID":"1441f96d-52b2-420c-bb88-988f13c65e65","Type":"ContainerStarted","Data":"fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d"}
Nov 03 11:00:25.870457 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.870314    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:25.870457 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.870340    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns" event={"ID":"1441f96d-52b2-420c-bb88-988f13c65e65","Type":"ContainerStarted","Data":"396ad3359d0e9f19342296f7eda678125bc8b7e52c7f21d96eb253243eae0d5f"}
Nov 03 11:00:25.887662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:25.887614    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns" podStartSLOduration=1.887599354 podStartE2EDuration="1.887599354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:25.885351795 +0000 UTC m=+1869.790206258" watchObservedRunningTime="2026-11-03 11:00:25.887599354 +0000 UTC m=+1869.792453817"
Nov 03 11:00:29.224205 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.224169    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"]
Nov 03 11:00:29.224770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.224758    5393 cpu_manager.go:414] "containerMap: removing container" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:29.224810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.224772    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676e924d-1a19-405c-a7d9-0489ccd9dccd" containerName="cleanup"
Nov 03 11:00:29.224810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.224800    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:29.224810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.224806    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:29.224957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.224826    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:29.224957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.224832    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be96e79-cf81-4f39-a1e3-b987253db254" containerName="cleanup"
Nov 03 11:00:29.224957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.224938    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbac1ba-fad2-48f6-a5e9-0c38df672ca2" containerName="cleanup"
Nov 03 11:00:29.228738 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.228720    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.233562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.233500    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:00:29.233562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.233564    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:00:29.236209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.236178    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"]
Nov 03 11:00:29.324145 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.324107    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.324293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.324209    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xmb\" (UniqueName: \"kubernetes.io/projected/dc23d6b3-b0e5-416d-985e-4de495d1231d-kube-api-access-p4xmb\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.324382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.324362    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.425587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.425551    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.425823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.425633    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xmb\" (UniqueName: \"kubernetes.io/projected/dc23d6b3-b0e5-416d-985e-4de495d1231d-kube-api-access-p4xmb\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.425898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.425849    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.426776 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.426747    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.428390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.428366    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.433887 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.433864    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xmb\" (UniqueName: \"kubernetes.io/projected/dc23d6b3-b0e5-416d-985e-4de495d1231d-kube-api-access-p4xmb\") pod \"maas-api-revoke-sub-36e85229-kx946\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.551966 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.551878    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:29.835472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.835438    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"]
Nov 03 11:00:29.837643 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:00:29.837611    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc23d6b3_b0e5_416d_985e_4de495d1231d.slice/crio-34af6aa81aeb04d4d7e018c7f4404cd2c1a6136143b610b58093f910302c39c1 WatchSource:0}: Error finding container 34af6aa81aeb04d4d7e018c7f4404cd2c1a6136143b610b58093f910302c39c1: Status 404 returned error can't find the container with id 34af6aa81aeb04d4d7e018c7f4404cd2c1a6136143b610b58093f910302c39c1
Nov 03 11:00:29.901466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:29.901433    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946" event={"ID":"dc23d6b3-b0e5-416d-985e-4de495d1231d","Type":"ContainerStarted","Data":"34af6aa81aeb04d4d7e018c7f4404cd2c1a6136143b610b58093f910302c39c1"}
Nov 03 11:00:30.911034 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:30.910999    5393 generic.go:356] "Generic (PLEG): container finished" podID="dc23d6b3-b0e5-416d-985e-4de495d1231d" containerID="7dd8e0341013ba875b8ddbc4eb166c264b2aef804c81198c8906c13d9488781d" exitCode=0
Nov 03 11:00:30.911436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:30.911090    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946" event={"ID":"dc23d6b3-b0e5-416d-985e-4de495d1231d","Type":"ContainerDied","Data":"7dd8e0341013ba875b8ddbc4eb166c264b2aef804c81198c8906c13d9488781d"}
Nov 03 11:00:32.183244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.183176    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:32.297186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.297151    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-openshift-ca\") pod \"dc23d6b3-b0e5-416d-985e-4de495d1231d\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") "
Nov 03 11:00:32.297367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.297335    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-xks-ca\") pod \"dc23d6b3-b0e5-416d-985e-4de495d1231d\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") "
Nov 03 11:00:32.297493 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.297482    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc23d6b3-b0e5-416d-985e-4de495d1231d-kube-api-access-p4xmb\" (UniqueName: \"kubernetes.io/projected/dc23d6b3-b0e5-416d-985e-4de495d1231d-kube-api-access-p4xmb\") pod \"dc23d6b3-b0e5-416d-985e-4de495d1231d\" (UID: \"dc23d6b3-b0e5-416d-985e-4de495d1231d\") "
Nov 03 11:00:32.297545 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.297521    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-openshift-ca" pod "dc23d6b3-b0e5-416d-985e-4de495d1231d" (UID: "dc23d6b3-b0e5-416d-985e-4de495d1231d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:00:32.297931 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.297897    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:32.299563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.299539    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-xks-ca" pod "dc23d6b3-b0e5-416d-985e-4de495d1231d" (UID: "dc23d6b3-b0e5-416d-985e-4de495d1231d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:00:32.299667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.299602    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc23d6b3-b0e5-416d-985e-4de495d1231d-kube-api-access-p4xmb" pod "dc23d6b3-b0e5-416d-985e-4de495d1231d" (UID: "dc23d6b3-b0e5-416d-985e-4de495d1231d"). InnerVolumeSpecName "kube-api-access-p4xmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:32.399318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.399284    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xmb\" (UniqueName: \"kubernetes.io/projected/dc23d6b3-b0e5-416d-985e-4de495d1231d-kube-api-access-p4xmb\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:32.399318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.399316    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc23d6b3-b0e5-416d-985e-4de495d1231d-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:32.927003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.926969    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946" event={"ID":"dc23d6b3-b0e5-416d-985e-4de495d1231d","Type":"ContainerDied","Data":"34af6aa81aeb04d4d7e018c7f4404cd2c1a6136143b610b58093f910302c39c1"}
Nov 03 11:00:32.927003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.926992    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"
Nov 03 11:00:32.927230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:32.927022    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34af6aa81aeb04d4d7e018c7f4404cd2c1a6136143b610b58093f910302c39c1"
Nov 03 11:00:35.022902 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:35.022869    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"]
Nov 03 11:00:35.028541 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:35.028509    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0080ac5-kx82p"]
Nov 03 11:00:36.233179 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:36.233149    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071535ef-1acc-4c69-ba0c-0d4785cb97b1" path="/var/lib/kubelet/pods/071535ef-1acc-4c69-ba0c-0d4785cb97b1/volumes"
Nov 03 11:00:36.880529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:36.880500    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:00:39.532550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:39.532504    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"]
Nov 03 11:00:43.192785 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:43.192740    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:00:47.610876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:47.610820    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"]
Nov 03 11:00:49.819675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.819629    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"]
Nov 03 11:00:49.820570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.820488    5393 cpu_manager.go:414] "containerMap: removing container" podUID="dc23d6b3-b0e5-416d-985e-4de495d1231d" containerName="revoke-keys"
Nov 03 11:00:49.820570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.820510    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc23d6b3-b0e5-416d-985e-4de495d1231d" containerName="revoke-keys"
Nov 03 11:00:49.820777 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.820628    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc23d6b3-b0e5-416d-985e-4de495d1231d" containerName="revoke-keys"
Nov 03 11:00:49.824816 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.824793    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.831108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.831078    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"]
Nov 03 11:00:49.833050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.833025    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-917da2-openshift-default-dockercfg-zw2p5\""
Nov 03 11:00:49.990987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.990942    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-envoy\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.993420 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.993392    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54s6n\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-kube-api-access-54s6n\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.993713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.993693    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istiod-ca-cert\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.993981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.993963    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-token\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.994173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.994157    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.994299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.994285    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-credential-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.994457 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.994440    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-podinfo\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.994611 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.994594    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-data\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:49.994737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:49.994724    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-certs\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.096874    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-envoy\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.096974    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54s6n\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-kube-api-access-54s6n\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.097023    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istiod-ca-cert\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.097157    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-token\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.097238    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.097283    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-credential-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.097357    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-podinfo\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.097421    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-data\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.097479    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-certs\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.098125    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-certs\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.098718    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.101994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.099675    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istiod-ca-cert\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.108375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.108309    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-data\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.116817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.116778    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-credential-socket\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.155044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.155006    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-podinfo\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.155800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.155768    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-envoy\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.168265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.168234    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-token\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.179035 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.170023    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54s6n\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-kube-api-access-54s6n\") pod \"e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") " pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.449010 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.448909    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:50.744538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.744502    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"]
Nov 03 11:00:50.744972 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:00:50.744946    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432522bc_6cbe_4cb5_a30c_9f9fad1b6be5.slice/crio-162c9806e9afe92a625aaa6e6c4075219831caf0d8b74048655b9b6f7c1b5e3e WatchSource:0}: Error finding container 162c9806e9afe92a625aaa6e6c4075219831caf0d8b74048655b9b6f7c1b5e3e: Status 404 returned error can't find the container with id 162c9806e9afe92a625aaa6e6c4075219831caf0d8b74048655b9b6f7c1b5e3e
Nov 03 11:00:50.747952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.747886    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:50.748043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.747974    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:50.748043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.748006    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:50.922443 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.922404    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"]
Nov 03 11:00:50.957190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.957164    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"]
Nov 03 11:00:50.957549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.957479    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:50.966521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:50.966493    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-56687c-openshift-default-dockercfg-flg5x\""
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236189    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-credential-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236235    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9215b-e2bd-4992-a6d6-97add590eb3b-istiod-ca-cert\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236295    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-token\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236445    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-envoy\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236539    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236602    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-podinfo\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236658    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-data\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236695    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj2kx\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-kube-api-access-gj2kx\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.262018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.236765    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-certs\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.348957    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-podinfo\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.349037    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-data\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.349069    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2kx\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-kube-api-access-gj2kx\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.349137    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-certs\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.349217    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-credential-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.349238    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9215b-e2bd-4992-a6d6-97add590eb3b-istiod-ca-cert\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.349287    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-token\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.349470    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-envoy\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.349590    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.350945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.350171    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.378941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.360482    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-credential-socket\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.378941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.360979    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-data\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.378941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.361656    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-certs\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.378941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.362509    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9215b-e2bd-4992-a6d6-97add590eb3b-istiod-ca-cert\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.405967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.396189    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" event={"ID":"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5","Type":"ContainerStarted","Data":"004134a50ed4b5e5d0e2b8d95468203ce976a3411605a9b0135306af0b8eeec6"}
Nov 03 11:00:51.405967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.396260    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" event={"ID":"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5","Type":"ContainerStarted","Data":"162c9806e9afe92a625aaa6e6c4075219831caf0d8b74048655b9b6f7c1b5e3e"}
Nov 03 11:00:51.413957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.413685    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-podinfo\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.414739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.414674    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-token\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.438442 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.436283    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-envoy\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.455258 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.455168    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:51.473954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.460299    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2kx\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-kube-api-access-gj2kx\") pod \"e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") " pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:51.477065 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.474208    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 11:00:51.477065 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.474271    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 11:00:51.599069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:51.598321    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:52.455103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:52.455076    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:52.475515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:52.475451    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podStartSLOduration=3.475432034 podStartE2EDuration="3.475432034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:51.436688586 +0000 UTC m=+1895.341543028" watchObservedRunningTime="2026-11-03 11:00:52.475432034 +0000 UTC m=+1896.380286480"
Nov 03 11:00:52.478606 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:52.478585    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"]
Nov 03 11:00:52.481182 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:00:52.481150    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a9215b_e2bd_4992_a6d6_97add590eb3b.slice/crio-9fef78212643b6aa62e419ec492d2942cd1ab075bc077223397e47c5db88bce1 WatchSource:0}: Error finding container 9fef78212643b6aa62e419ec492d2942cd1ab075bc077223397e47c5db88bce1: Status 404 returned error can't find the container with id 9fef78212643b6aa62e419ec492d2942cd1ab075bc077223397e47c5db88bce1
Nov 03 11:00:52.484790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:52.484750    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:52.484881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:52.484839    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:52.484941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:52.484883    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:00:53.435306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:53.435266    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" event={"ID":"41a9215b-e2bd-4992-a6d6-97add590eb3b","Type":"ContainerStarted","Data":"74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e"}
Nov 03 11:00:53.435580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:53.435559    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" event={"ID":"41a9215b-e2bd-4992-a6d6-97add590eb3b","Type":"ContainerStarted","Data":"9fef78212643b6aa62e419ec492d2942cd1ab075bc077223397e47c5db88bce1"}
Nov 03 11:00:53.436093 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:53.436071    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:53.438449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:53.438429    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:00:53.460892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:53.460825    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podStartSLOduration=3.460806689 podStartE2EDuration="3.460806689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:53.457313731 +0000 UTC m=+1897.362168173" watchObservedRunningTime="2026-11-03 11:00:53.460806689 +0000 UTC m=+1897.365661133"
Nov 03 11:00:53.620947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:53.619001    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:53.639944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:53.637857    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:54.442752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:54.442716    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:54.443540 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:54.443520    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:00:54.582944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:54.576277    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f" podUID="e1155f79-030f-4be9-8ce5-5123a4f936c2" containerName="main" containerID="cri-o://653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc" gracePeriod=45
Nov 03 11:00:54.927242 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:54.927217    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 11:00:55.012512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.012418    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1155f79-030f-4be9-8ce5-5123a4f936c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1155f79-030f-4be9-8ce5-5123a4f936c2-tls-certs\") pod \"e1155f79-030f-4be9-8ce5-5123a4f936c2\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") "
Nov 03 11:00:55.012512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.012504    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1155f79-030f-4be9-8ce5-5123a4f936c2-kube-api-access-9zmqb\" (UniqueName: \"kubernetes.io/projected/e1155f79-030f-4be9-8ce5-5123a4f936c2-kube-api-access-9zmqb\") pod \"e1155f79-030f-4be9-8ce5-5123a4f936c2\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") "
Nov 03 11:00:55.012727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.012581    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-home\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-home\") pod \"e1155f79-030f-4be9-8ce5-5123a4f936c2\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") "
Nov 03 11:00:55.012854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.012837    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-model-cache\") pod \"e1155f79-030f-4be9-8ce5-5123a4f936c2\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") "
Nov 03 11:00:55.012967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.012873    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-dshm\") pod \"e1155f79-030f-4be9-8ce5-5123a4f936c2\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") "
Nov 03 11:00:55.012967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.012956    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-tmp-dir\") pod \"e1155f79-030f-4be9-8ce5-5123a4f936c2\" (UID: \"e1155f79-030f-4be9-8ce5-5123a4f936c2\") "
Nov 03 11:00:55.013650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.013470    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-model-cache" pod "e1155f79-030f-4be9-8ce5-5123a4f936c2" (UID: "e1155f79-030f-4be9-8ce5-5123a4f936c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:00:55.013774 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.013697    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-tmp-dir" pod "e1155f79-030f-4be9-8ce5-5123a4f936c2" (UID: "e1155f79-030f-4be9-8ce5-5123a4f936c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:00:55.013881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.013863    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:55.013971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.013871    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-home" pod "e1155f79-030f-4be9-8ce5-5123a4f936c2" (UID: "e1155f79-030f-4be9-8ce5-5123a4f936c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:00:55.016295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.016271    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-dshm" pod "e1155f79-030f-4be9-8ce5-5123a4f936c2" (UID: "e1155f79-030f-4be9-8ce5-5123a4f936c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:00:55.029500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.029454    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1155f79-030f-4be9-8ce5-5123a4f936c2-tls-certs" pod "e1155f79-030f-4be9-8ce5-5123a4f936c2" (UID: "e1155f79-030f-4be9-8ce5-5123a4f936c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:00:55.031081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.031058    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1155f79-030f-4be9-8ce5-5123a4f936c2-kube-api-access-9zmqb" pod "e1155f79-030f-4be9-8ce5-5123a4f936c2" (UID: "e1155f79-030f-4be9-8ce5-5123a4f936c2"). InnerVolumeSpecName "kube-api-access-9zmqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:00:55.117120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.116204    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:55.117120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.116247    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:55.117120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.116263    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1155f79-030f-4be9-8ce5-5123a4f936c2-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:55.117120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.116278    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zmqb\" (UniqueName: \"kubernetes.io/projected/e1155f79-030f-4be9-8ce5-5123a4f936c2-kube-api-access-9zmqb\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:55.117120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.116292    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1155f79-030f-4be9-8ce5-5123a4f936c2-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:00:55.276896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.276794    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"]
Nov 03 11:00:55.277756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.277721    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e1155f79-030f-4be9-8ce5-5123a4f936c2" containerName="main"
Nov 03 11:00:55.277756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.277756    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1155f79-030f-4be9-8ce5-5123a4f936c2" containerName="main"
Nov 03 11:00:55.277995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.277985    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1155f79-030f-4be9-8ce5-5123a4f936c2" containerName="main"
Nov 03 11:00:55.281904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.281876    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:55.307990 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.305590    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"]
Nov 03 11:00:55.315596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.315567    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-917da2\""
Nov 03 11:00:55.319674 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.319639    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhjs\" (UniqueName: \"kubernetes.io/projected/b2aa0412-2a7f-4850-ba77-92f35382670f-kube-api-access-tfhjs\") pod \"payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t\" (UID: \"b2aa0412-2a7f-4850-ba77-92f35382670f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:55.320663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.320031    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2aa0412-2a7f-4850-ba77-92f35382670f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t\" (UID: \"b2aa0412-2a7f-4850-ba77-92f35382670f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:55.358804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.358762    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-917da2-dockercfg-cdql9\""
Nov 03 11:00:55.470945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.423011    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2aa0412-2a7f-4850-ba77-92f35382670f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t\" (UID: \"b2aa0412-2a7f-4850-ba77-92f35382670f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:55.471510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.471475    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhjs\" (UniqueName: \"kubernetes.io/projected/b2aa0412-2a7f-4850-ba77-92f35382670f-kube-api-access-tfhjs\") pod \"payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t\" (UID: \"b2aa0412-2a7f-4850-ba77-92f35382670f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:55.514941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.460762    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2aa0412-2a7f-4850-ba77-92f35382670f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t\" (UID: \"b2aa0412-2a7f-4850-ba77-92f35382670f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:55.621994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.584671    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhjs\" (UniqueName: \"kubernetes.io/projected/b2aa0412-2a7f-4850-ba77-92f35382670f-kube-api-access-tfhjs\") pod \"payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t\" (UID: \"b2aa0412-2a7f-4850-ba77-92f35382670f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:55.642951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.635150    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:55.711942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.704674    5393 generic.go:356] "Generic (PLEG): container finished" podID="e1155f79-030f-4be9-8ce5-5123a4f936c2" containerID="653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc" exitCode=0
Nov 03 11:00:55.711942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.705721    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"
Nov 03 11:00:55.732952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.730228    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f" event={"ID":"e1155f79-030f-4be9-8ce5-5123a4f936c2","Type":"ContainerDied","Data":"653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc"}
Nov 03 11:00:55.732952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.730297    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f" event={"ID":"e1155f79-030f-4be9-8ce5-5123a4f936c2","Type":"ContainerDied","Data":"4e858107c55336eeec84d322c577c3df6cbaa090dd991c46111d9782da4c4054"}
Nov 03 11:00:55.732952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.730323    5393 scope.go:122] "RemoveContainer" containerID="653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc"
Nov 03 11:00:55.780795 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.780708    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"]
Nov 03 11:00:55.786315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.786282    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/auth-test-model-6012b5-kserve-868ff7649-9bh2f"]
Nov 03 11:00:55.819166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.804359    5393 scope.go:122] "RemoveContainer" containerID="653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc"
Nov 03 11:00:55.819166 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:00:55.806182    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc\": container with ID starting with 653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc not found: ID does not exist" containerID="653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc"
Nov 03 11:00:55.819166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:55.806240    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc"} err="failed to get container status \"653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc\": rpc error: code = NotFound desc = could not find container \"653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc\": container with ID starting with 653e7cbeed4c20a74a32fad7cefad1c59aaa3b3286f051b2a43ea6127fc7d9fc not found: ID does not exist"
Nov 03 11:00:56.095941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.089108    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"]
Nov 03 11:00:56.234939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.234890    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1155f79-030f-4be9-8ce5-5123a4f936c2" path="/var/lib/kubelet/pods/e1155f79-030f-4be9-8ce5-5123a4f936c2/volumes"
Nov 03 11:00:56.714748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.714697    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t" event={"ID":"b2aa0412-2a7f-4850-ba77-92f35382670f","Type":"ContainerStarted","Data":"64700768101b9dfc287508e08ff3e164baf2f75080c663f0f7a68b65892aaf27"}
Nov 03 11:00:56.715343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.714769    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t" event={"ID":"b2aa0412-2a7f-4850-ba77-92f35382670f","Type":"ContainerStarted","Data":"ace17dcc5f014d5284f21abd40316d030f2a4cd3b6ce435b91dd794eb9b460aa"}
Nov 03 11:00:56.715969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.715937    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:00:56.733359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.733296    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t" podStartSLOduration=1.7332768280000002 podStartE2EDuration="1.733276828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:56.730735623 +0000 UTC m=+1900.635590068" watchObservedRunningTime="2026-11-03 11:00:56.733276828 +0000 UTC m=+1900.638131270"
Nov 03 11:00:56.763652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.763617    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"]
Nov 03 11:00:56.782965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.782888    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"]
Nov 03 11:00:56.783142 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.783021    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:56.789322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.789286    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cbeebd\""
Nov 03 11:00:56.798304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.798272    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cbeebd-dockercfg-bb2mp\""
Nov 03 11:00:56.863091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.863055    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eda48e-713c-4144-8608-b2f7a2f56503-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6\" (UID: \"18eda48e-713c-4144-8608-b2f7a2f56503\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:56.863259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.863233    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4c2\" (UniqueName: \"kubernetes.io/projected/18eda48e-713c-4144-8608-b2f7a2f56503-kube-api-access-fs4c2\") pod \"payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6\" (UID: \"18eda48e-713c-4144-8608-b2f7a2f56503\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:56.970998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.965068    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4c2\" (UniqueName: \"kubernetes.io/projected/18eda48e-713c-4144-8608-b2f7a2f56503-kube-api-access-fs4c2\") pod \"payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6\" (UID: \"18eda48e-713c-4144-8608-b2f7a2f56503\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:56.970998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.965326    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eda48e-713c-4144-8608-b2f7a2f56503-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6\" (UID: \"18eda48e-713c-4144-8608-b2f7a2f56503\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:56.997956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:56.991980    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eda48e-713c-4144-8608-b2f7a2f56503-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6\" (UID: \"18eda48e-713c-4144-8608-b2f7a2f56503\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:57.010996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:57.010963    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4c2\" (UniqueName: \"kubernetes.io/projected/18eda48e-713c-4144-8608-b2f7a2f56503-kube-api-access-fs4c2\") pod \"payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6\" (UID: \"18eda48e-713c-4144-8608-b2f7a2f56503\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:57.104947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:57.104182    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:57.569420 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:00:57.569387    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18eda48e_713c_4144_8608_b2f7a2f56503.slice/crio-ba50dc607e02a5892cd4b23af7a3c0ce253c4faf8f12c148fafdafbc9950e0c4 WatchSource:0}: Error finding container ba50dc607e02a5892cd4b23af7a3c0ce253c4faf8f12c148fafdafbc9950e0c4: Status 404 returned error can't find the container with id ba50dc607e02a5892cd4b23af7a3c0ce253c4faf8f12c148fafdafbc9950e0c4
Nov 03 11:00:57.569801 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:57.569778    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"]
Nov 03 11:00:57.795189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:57.795150    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6" event={"ID":"18eda48e-713c-4144-8608-b2f7a2f56503","Type":"ContainerStarted","Data":"347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c"}
Nov 03 11:00:57.795749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:57.795215    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6" event={"ID":"18eda48e-713c-4144-8608-b2f7a2f56503","Type":"ContainerStarted","Data":"ba50dc607e02a5892cd4b23af7a3c0ce253c4faf8f12c148fafdafbc9950e0c4"}
Nov 03 11:00:57.795749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:57.795411    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:00:57.817827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:00:57.817770    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6" podStartSLOduration=1.817756272 podStartE2EDuration="1.817756272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:00:57.814653502 +0000 UTC m=+1901.719507943" watchObservedRunningTime="2026-11-03 11:00:57.817756272 +0000 UTC m=+1901.722610713"
Nov 03 11:01:02.640063 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:02.640026    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k" podUID="e0819cc7-284b-45b2-865a-02e278372617" containerName="main" containerID="cri-o://4f6bc482a3431d851e65007257865517315c76ad10468ca2c7548159f7cc2858" gracePeriod=45
Nov 03 11:01:02.856083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:02.855798    5393 generic.go:356] "Generic (PLEG): container finished" podID="e0819cc7-284b-45b2-865a-02e278372617" containerID="4f6bc482a3431d851e65007257865517315c76ad10468ca2c7548159f7cc2858" exitCode=0
Nov 03 11:01:02.856083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:02.856034    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k" event={"ID":"e0819cc7-284b-45b2-865a-02e278372617","Type":"ContainerDied","Data":"4f6bc482a3431d851e65007257865517315c76ad10468ca2c7548159f7cc2858"}
Nov 03 11:01:02.928161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:02.928132    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 11:01:03.005411 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.005380    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-home\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-home\") pod \"e0819cc7-284b-45b2-865a-02e278372617\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") "
Nov 03 11:01:03.006137 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.006110    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-model-cache\") pod \"e0819cc7-284b-45b2-865a-02e278372617\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") "
Nov 03 11:01:03.007172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.007151    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-home" pod "e0819cc7-284b-45b2-865a-02e278372617" (UID: "e0819cc7-284b-45b2-865a-02e278372617"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:03.008194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.008171    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-model-cache" pod "e0819cc7-284b-45b2-865a-02e278372617" (UID: "e0819cc7-284b-45b2-865a-02e278372617"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:03.107541 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.107505    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0819cc7-284b-45b2-865a-02e278372617-kube-api-access-hhcm5\" (UniqueName: \"kubernetes.io/projected/e0819cc7-284b-45b2-865a-02e278372617-kube-api-access-hhcm5\") pod \"e0819cc7-284b-45b2-865a-02e278372617\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") "
Nov 03 11:01:03.107749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.107562    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-dshm\") pod \"e0819cc7-284b-45b2-865a-02e278372617\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") "
Nov 03 11:01:03.107749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.107700    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-tmp-dir\") pod \"e0819cc7-284b-45b2-865a-02e278372617\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") "
Nov 03 11:01:03.107749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.107740    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0819cc7-284b-45b2-865a-02e278372617-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0819cc7-284b-45b2-865a-02e278372617-tls-certs\") pod \"e0819cc7-284b-45b2-865a-02e278372617\" (UID: \"e0819cc7-284b-45b2-865a-02e278372617\") "
Nov 03 11:01:03.108022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.107994    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-tmp-dir" pod "e0819cc7-284b-45b2-865a-02e278372617" (UID: "e0819cc7-284b-45b2-865a-02e278372617"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:03.108395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.108373    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:03.108395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.108394    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:03.108595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.108402    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:03.110276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.110247    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-dshm" pod "e0819cc7-284b-45b2-865a-02e278372617" (UID: "e0819cc7-284b-45b2-865a-02e278372617"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:03.110405 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.110275    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0819cc7-284b-45b2-865a-02e278372617-tls-certs" pod "e0819cc7-284b-45b2-865a-02e278372617" (UID: "e0819cc7-284b-45b2-865a-02e278372617"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:01:03.111392 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.111370    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0819cc7-284b-45b2-865a-02e278372617-kube-api-access-hhcm5" pod "e0819cc7-284b-45b2-865a-02e278372617" (UID: "e0819cc7-284b-45b2-865a-02e278372617"). InnerVolumeSpecName "kube-api-access-hhcm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:01:03.209510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.209416    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhcm5\" (UniqueName: \"kubernetes.io/projected/e0819cc7-284b-45b2-865a-02e278372617-kube-api-access-hhcm5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:03.209510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.209450    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0819cc7-284b-45b2-865a-02e278372617-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:03.209510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.209461    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0819cc7-284b-45b2-865a-02e278372617-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:03.866229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.866176    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k" event={"ID":"e0819cc7-284b-45b2-865a-02e278372617","Type":"ContainerDied","Data":"2f6f85cdfad8a34188acecd0816eb33fdb5e6f516982bbb9d855158b7dadd69c"}
Nov 03 11:01:03.866670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.866244    5393 scope.go:122] "RemoveContainer" containerID="4f6bc482a3431d851e65007257865517315c76ad10468ca2c7548159f7cc2858"
Nov 03 11:01:03.866670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.866407    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"
Nov 03 11:01:03.893210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.893180    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"]
Nov 03 11:01:03.896491 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:03.896473    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/auth-test-model-cc8214-kserve-68d5cc7bd9-7kz5k"]
Nov 03 11:01:04.232351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:04.232269    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0819cc7-284b-45b2-865a-02e278372617" path="/var/lib/kubelet/pods/e0819cc7-284b-45b2-865a-02e278372617/volumes"
Nov 03 11:01:07.213373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.213334    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"]
Nov 03 11:01:07.214018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.214002    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e0819cc7-284b-45b2-865a-02e278372617" containerName="main"
Nov 03 11:01:07.214082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.214019    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0819cc7-284b-45b2-865a-02e278372617" containerName="main"
Nov 03 11:01:07.214171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.214162    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0819cc7-284b-45b2-865a-02e278372617" containerName="main"
Nov 03 11:01:07.219137 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.219105    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:07.223214 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.223188    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-56687c\""
Nov 03 11:01:07.227529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.227505    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"]
Nov 03 11:01:07.248037 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.248006    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"]
Nov 03 11:01:07.257225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.257195    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:07.258040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.258011    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"]
Nov 03 11:01:07.260816 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.260792    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-56687c-dockercfg-jsqv9\""
Nov 03 11:01:07.308116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.308078    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfw9\" (UniqueName: \"kubernetes.io/projected/f3a3de92-9886-41bd-99ab-2fa5ad62b841-kube-api-access-6tfw9\") pod \"payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws\" (UID: \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:07.308314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.308174    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56qq\" (UniqueName: \"kubernetes.io/projected/b087ec3a-d711-4e4f-b219-c6d026fa08c4-kube-api-access-q56qq\") pod \"payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk\" (UID: \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:07.308314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.308248    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b087ec3a-d711-4e4f-b219-c6d026fa08c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk\" (UID: \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:07.308791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.308467    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a3de92-9886-41bd-99ab-2fa5ad62b841-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws\" (UID: \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:07.409674 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.409632    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a3de92-9886-41bd-99ab-2fa5ad62b841-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws\" (UID: \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:07.409878 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.409742    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfw9\" (UniqueName: \"kubernetes.io/projected/f3a3de92-9886-41bd-99ab-2fa5ad62b841-kube-api-access-6tfw9\") pod \"payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws\" (UID: \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:07.409878 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.409813    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56qq\" (UniqueName: \"kubernetes.io/projected/b087ec3a-d711-4e4f-b219-c6d026fa08c4-kube-api-access-q56qq\") pod \"payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk\" (UID: \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:07.410027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.409886    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b087ec3a-d711-4e4f-b219-c6d026fa08c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk\" (UID: \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:07.410572 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.410548    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a3de92-9886-41bd-99ab-2fa5ad62b841-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws\" (UID: \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:07.410745 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.410724    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b087ec3a-d711-4e4f-b219-c6d026fa08c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk\" (UID: \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:07.418175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.418136    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfw9\" (UniqueName: \"kubernetes.io/projected/f3a3de92-9886-41bd-99ab-2fa5ad62b841-kube-api-access-6tfw9\") pod \"payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws\" (UID: \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:07.418175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.418167    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56qq\" (UniqueName: \"kubernetes.io/projected/b087ec3a-d711-4e4f-b219-c6d026fa08c4-kube-api-access-q56qq\") pod \"payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk\" (UID: \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:07.540278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.540186    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:07.597903 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:07.597873    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:08.460061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:08.460025    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"]
Nov 03 11:01:08.466536 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:01:08.466505    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb087ec3a_d711_4e4f_b219_c6d026fa08c4.slice/crio-6fad796d6bb7056a7fa7b57b367019cf861cc8f34d8ad9b7cc8fbcd77785c4c3 WatchSource:0}: Error finding container 6fad796d6bb7056a7fa7b57b367019cf861cc8f34d8ad9b7cc8fbcd77785c4c3: Status 404 returned error can't find the container with id 6fad796d6bb7056a7fa7b57b367019cf861cc8f34d8ad9b7cc8fbcd77785c4c3
Nov 03 11:01:08.499950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:08.497830    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"]
Nov 03 11:01:08.807435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:08.807345    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:01:08.807435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:08.807424    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:01:09.140735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:09.139473    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk" event={"ID":"b087ec3a-d711-4e4f-b219-c6d026fa08c4","Type":"ContainerStarted","Data":"ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4"}
Nov 03 11:01:09.140735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:09.139541    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk" event={"ID":"b087ec3a-d711-4e4f-b219-c6d026fa08c4","Type":"ContainerStarted","Data":"6fad796d6bb7056a7fa7b57b367019cf861cc8f34d8ad9b7cc8fbcd77785c4c3"}
Nov 03 11:01:09.140735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:09.140598    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:09.143612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:09.143584    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws" event={"ID":"f3a3de92-9886-41bd-99ab-2fa5ad62b841","Type":"ContainerStarted","Data":"f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d"}
Nov 03 11:01:09.143839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:09.143819    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws" event={"ID":"f3a3de92-9886-41bd-99ab-2fa5ad62b841","Type":"ContainerStarted","Data":"24cf7ae75382ef73d80b99af46a649d5452373e83dc4c70809e078e7deeaaa6a"}
Nov 03 11:01:09.144947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:09.144907    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:09.164138 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:09.164085    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk" podStartSLOduration=2.164067148 podStartE2EDuration="2.164067148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:09.163042829 +0000 UTC m=+1913.067897274" watchObservedRunningTime="2026-11-03 11:01:09.164067148 +0000 UTC m=+1913.068921591"
Nov 03 11:01:09.186427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:09.186369    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws" podStartSLOduration=2.186351693 podStartE2EDuration="2.186351693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:09.182018559 +0000 UTC m=+1913.086873097" watchObservedRunningTime="2026-11-03 11:01:09.186351693 +0000 UTC m=+1913.091206134"
Nov 03 11:01:13.191576 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:01:13.191538    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:01:18.256082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:18.256041    5393 scope.go:122] "RemoveContainer" containerID="3a2ab793137783c741d85120d1963e8a7bf34ad831eaf3e93de561922cc25a9a"
Nov 03 11:01:18.266619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:18.266591    5393 scope.go:122] "RemoveContainer" containerID="445f832df449b92a4c733c647b44a2b762ab2c33f962cfc56e797897923db830"
Nov 03 11:01:21.207946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:21.203139    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:01:21.221944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:21.219100    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:01:28.984795 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:28.984700    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"]
Nov 03 11:01:28.985422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:28.984990    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy" containerID="cri-o://dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20" gracePeriod=30
Nov 03 11:01:28.987024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:28.986997    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"]
Nov 03 11:01:28.987247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:28.987223    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy" containerID="cri-o://b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8" gracePeriod=30
Nov 03 11:01:28.992170 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:28.992131    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"]
Nov 03 11:01:28.992359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:28.992335    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy" containerID="cri-o://041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8" gracePeriod=30
Nov 03 11:01:31.319648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.319615    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"]
Nov 03 11:01:31.326188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.326164    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:31.330373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.330341    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fb5766\""
Nov 03 11:01:31.331807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.331782    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fb5766-dockercfg-vcrpk\""
Nov 03 11:01:31.332415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.332360    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"]
Nov 03 11:01:31.346035 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.346005    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"]
Nov 03 11:01:31.351676 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.351651    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:31.357139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.357089    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"]
Nov 03 11:01:31.405854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.405818    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvthc\" (UniqueName: \"kubernetes.io/projected/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-kube-api-access-bvthc\") pod \"payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl\" (UID: \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:31.406095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.405883    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl\" (UID: \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:31.540006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.538347    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvthc\" (UniqueName: \"kubernetes.io/projected/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-kube-api-access-bvthc\") pod \"payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl\" (UID: \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:31.540006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.538423    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmrmh\" (UniqueName: \"kubernetes.io/projected/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-kube-api-access-rmrmh\") pod \"payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh\" (UID: \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:31.540006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.538477    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl\" (UID: \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:31.540006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.538728    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh\" (UID: \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:31.540387 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.540026    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl\" (UID: \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:31.642944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.624842    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvthc\" (UniqueName: \"kubernetes.io/projected/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-kube-api-access-bvthc\") pod \"payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl\" (UID: \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:31.643158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.643030    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh\" (UID: \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:31.643231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.643182    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmrmh\" (UniqueName: \"kubernetes.io/projected/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-kube-api-access-rmrmh\") pod \"payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh\" (UID: \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:31.655938 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.644476    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh\" (UID: \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:31.668446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.667225    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:31.671947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.669737    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmrmh\" (UniqueName: \"kubernetes.io/projected/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-kube-api-access-rmrmh\") pod \"payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh\" (UID: \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:31.704907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:31.684203    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:32.039851 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.039828    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"]
Nov 03 11:01:32.040312 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:01:32.040290    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5cec20_8d50_44f1_b626_b60ce1ca5c3b.slice/crio-a6f2b30e83b6b82b1366923f88f66bf26f9b2105ade0294f5658f60a66aa4713 WatchSource:0}: Error finding container a6f2b30e83b6b82b1366923f88f66bf26f9b2105ade0294f5658f60a66aa4713: Status 404 returned error can't find the container with id a6f2b30e83b6b82b1366923f88f66bf26f9b2105ade0294f5658f60a66aa4713
Nov 03 11:01:32.069664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.069638    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"]
Nov 03 11:01:32.081850 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:01:32.081826    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b2ea3b_ee96_4d6d_9a9b_8721d5e6c0bc.slice/crio-92306c1aa5019a4694aaa58e3ce428c5e567d85070a81040278f52971214ff59 WatchSource:0}: Error finding container 92306c1aa5019a4694aaa58e3ce428c5e567d85070a81040278f52971214ff59: Status 404 returned error can't find the container with id 92306c1aa5019a4694aaa58e3ce428c5e567d85070a81040278f52971214ff59
Nov 03 11:01:32.617818 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.617780    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh" event={"ID":"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc","Type":"ContainerStarted","Data":"db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17"}
Nov 03 11:01:32.618315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.617843    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh" event={"ID":"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc","Type":"ContainerStarted","Data":"92306c1aa5019a4694aaa58e3ce428c5e567d85070a81040278f52971214ff59"}
Nov 03 11:01:32.618315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.618080    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:32.619983 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.619951    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl" event={"ID":"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b","Type":"ContainerStarted","Data":"4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4"}
Nov 03 11:01:32.620140 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.620000    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl" event={"ID":"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b","Type":"ContainerStarted","Data":"a6f2b30e83b6b82b1366923f88f66bf26f9b2105ade0294f5658f60a66aa4713"}
Nov 03 11:01:32.620140 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.620044    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:32.635037 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.634980    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh" podStartSLOduration=1.634962287 podStartE2EDuration="1.634962287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:32.632120856 +0000 UTC m=+1936.536975302" watchObservedRunningTime="2026-11-03 11:01:32.634962287 +0000 UTC m=+1936.539816729"
Nov 03 11:01:32.650202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:32.650150    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl" podStartSLOduration=1.650135783 podStartE2EDuration="1.650135783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:32.647572666 +0000 UTC m=+1936.552427108" watchObservedRunningTime="2026-11-03 11:01:32.650135783 +0000 UTC m=+1936.554990224"
Nov 03 11:01:34.326051 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.325971    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 11:01:34.331300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.331278    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 11:01:34.364577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.364543    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 11:01:34.436348 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436316    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-socket\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.436579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436400    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-certs\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.436579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436457    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-envoy\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.436579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436506    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istiod-ca-cert\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.436579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436540    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-token\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-token\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.436579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436578    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-credential-socket\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.436890 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436649    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-kube-api-access-mgxg7\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-kube-api-access-mgxg7\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.436890 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436716    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-credential-socket\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.436890 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436747    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-data\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.436890 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436769    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-podinfo\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.436890 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436803    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-socket\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.436890 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436865    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-certs\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.437244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436955    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-podinfo\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.437244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436983    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-token\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.437244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.437010    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istiod-ca-cert\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.437244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.437042    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-kube-api-access-qbwfw\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-kube-api-access-qbwfw\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.437244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.437090    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-data\") pod \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\" (UID: \"ea18e571-df1d-4711-aa0c-1c11ca6be4c3\") "
Nov 03 11:01:34.437244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.437111    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-envoy\") pod \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\" (UID: \"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7\") "
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.436640    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-socket" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.437015    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istiod-ca-cert" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.437765    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-certs" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.438579    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-data" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.438638    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-data" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.438768    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-credential-socket" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.438854    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-credential-socket" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.438887    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-socket" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.438949    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-certs" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.439393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.439249    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istiod-ca-cert" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:01:34.442375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.442348    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-podinfo" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:01:34.442720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.442689    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-kube-api-access-mgxg7" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "kube-api-access-mgxg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:01:34.442720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.442689    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-podinfo" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:01:34.442842 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.442718    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-envoy" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.442842 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.442754    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-envoy" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.443684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.443656    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-kube-api-access-qbwfw" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "kube-api-access-qbwfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:01:34.443773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.443753    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-token" pod "ea18e571-df1d-4711-aa0c-1c11ca6be4c3" (UID: "ea18e571-df1d-4711-aa0c-1c11ca6be4c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:01:34.444228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.444205    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-token" pod "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" (UID: "d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.541767    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-envoy\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.541841    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-token\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-token\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.542043    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-data\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.542168    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-credential-socket\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.542215    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-kube-api-access-59dsn\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-kube-api-access-59dsn\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.542300    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-podinfo\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.542418    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-certs\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.542510    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da57fce0-41d2-4546-a567-7dcc3bdc9221-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da57fce0-41d2-4546-a567-7dcc3bdc9221-istiod-ca-cert\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.542561    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-socket\") pod \"da57fce0-41d2-4546-a567-7dcc3bdc9221\" (UID: \"da57fce0-41d2-4546-a567-7dcc3bdc9221\") "
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543442    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543468    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543486    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543500    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543513    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543530    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543545    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgxg7\" (UniqueName: \"kubernetes.io/projected/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-kube-api-access-mgxg7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543559    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543573    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543586    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543601    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543613    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543638    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543653    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543667    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543683    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbwfw\" (UniqueName: \"kubernetes.io/projected/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-kube-api-access-qbwfw\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543702    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18e571-df1d-4711-aa0c-1c11ca6be4c3-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.548002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.543719    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.576826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.567662    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-certs" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.576826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.569573    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da57fce0-41d2-4546-a567-7dcc3bdc9221-istiod-ca-cert" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:01:34.576826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.569637    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-podinfo" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:01:34.576826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.572345    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-token" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:01:34.576826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.573981    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-socket" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.576826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.576560    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-data" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.576826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.576673    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-credential-socket" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.577398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.576844    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-envoy" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:01:34.577398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.577018    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-kube-api-access-59dsn" pod "da57fce0-41d2-4546-a567-7dcc3bdc9221" (UID: "da57fce0-41d2-4546-a567-7dcc3bdc9221"). InnerVolumeSpecName "kube-api-access-59dsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:01:34.635185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.635152    5393 generic.go:356] "Generic (PLEG): container finished" podID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerID="dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20" exitCode=0
Nov 03 11:01:34.635388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.635199    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"
Nov 03 11:01:34.635388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.635238    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" event={"ID":"ea18e571-df1d-4711-aa0c-1c11ca6be4c3","Type":"ContainerDied","Data":"dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20"}
Nov 03 11:01:34.635388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.635296    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r" event={"ID":"ea18e571-df1d-4711-aa0c-1c11ca6be4c3","Type":"ContainerDied","Data":"288c2023a0bb5db13dfe2ee16d778186bbfdf720dbd31bd28d4d51159b328dd8"}
Nov 03 11:01:34.635388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.635329    5393 scope.go:122] "RemoveContainer" containerID="dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20"
Nov 03 11:01:34.636733 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.636710    5393 generic.go:356] "Generic (PLEG): container finished" podID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerID="041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8" exitCode=0
Nov 03 11:01:34.636855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.636767    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"
Nov 03 11:01:34.636855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.636796    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" event={"ID":"da57fce0-41d2-4546-a567-7dcc3bdc9221","Type":"ContainerDied","Data":"041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8"}
Nov 03 11:01:34.636855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.636838    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx" event={"ID":"da57fce0-41d2-4546-a567-7dcc3bdc9221","Type":"ContainerDied","Data":"45dfa602b1b3e394786c01746eae822ac9b4b82866d7c2c291e15263c22e9043"}
Nov 03 11:01:34.638508 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.638486    5393 generic.go:356] "Generic (PLEG): container finished" podID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerID="b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8" exitCode=0
Nov 03 11:01:34.638617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.638515    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" event={"ID":"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7","Type":"ContainerDied","Data":"b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8"}
Nov 03 11:01:34.638617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.638533    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"
Nov 03 11:01:34.638617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.638560    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk" event={"ID":"d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7","Type":"ContainerDied","Data":"9f50cec797cfd8b27499e9019ac5684aeea4b76865520c59b2b835a32237e059"}
Nov 03 11:01:34.644728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644703    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59dsn\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-kube-api-access-59dsn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.644728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644730    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.644892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644739    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.644892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644750    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da57fce0-41d2-4546-a567-7dcc3bdc9221-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.644892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644759    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.644892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644767    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.644892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644775    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.644892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644783    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.644892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.644790    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da57fce0-41d2-4546-a567-7dcc3bdc9221-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:34.648095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.648075    5393 scope.go:122] "RemoveContainer" containerID="dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20"
Nov 03 11:01:34.648346 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:01:34.648329    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20\": container with ID starting with dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20 not found: ID does not exist" containerID="dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20"
Nov 03 11:01:34.648430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.648355    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20"} err="failed to get container status \"dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20\": rpc error: code = NotFound desc = could not find container \"dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20\": container with ID starting with dc1e47673444908e4a7cb2005b15a8f609ee08ad131b7d205b13a5a488abfd20 not found: ID does not exist"
Nov 03 11:01:34.648430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.648375    5393 scope.go:122] "RemoveContainer" containerID="041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8"
Nov 03 11:01:34.658734 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.658718    5393 scope.go:122] "RemoveContainer" containerID="041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8"
Nov 03 11:01:34.659053 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:01:34.659034    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8\": container with ID starting with 041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8 not found: ID does not exist" containerID="041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8"
Nov 03 11:01:34.659116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.659060    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8"} err="failed to get container status \"041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8\": rpc error: code = NotFound desc = could not find container \"041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8\": container with ID starting with 041256c37bd439879a3a2e9ee102dcf66b5391f4930c500daa8a170d4c8c35e8 not found: ID does not exist"
Nov 03 11:01:34.659116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.659077    5393 scope.go:122] "RemoveContainer" containerID="b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8"
Nov 03 11:01:34.669666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.669641    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"]
Nov 03 11:01:34.672871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.672843    5393 scope.go:122] "RemoveContainer" containerID="b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8"
Nov 03 11:01:34.673624 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:01:34.673274    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8\": container with ID starting with b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8 not found: ID does not exist" containerID="b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8"
Nov 03 11:01:34.673624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.673329    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8"} err="failed to get container status \"b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8\": rpc error: code = NotFound desc = could not find container \"b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8\": container with ID starting with b12571e4b6d23b5307897d5e1906a9ae2648aa99bc7d39a2be70c27958e7c5d8 not found: ID does not exist"
Nov 03 11:01:34.675047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.675018    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-c5p2r"]
Nov 03 11:01:34.686101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.686075    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"]
Nov 03 11:01:34.689747 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.689717    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-ms8hx"]
Nov 03 11:01:34.700477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.700455    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"]
Nov 03 11:01:34.703755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:34.703735    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-8x9nk"]
Nov 03 11:01:36.239282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:36.239244    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" path="/var/lib/kubelet/pods/d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7/volumes"
Nov 03 11:01:36.239975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:36.239956    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" path="/var/lib/kubelet/pods/da57fce0-41d2-4546-a567-7dcc3bdc9221/volumes"
Nov 03 11:01:36.240561 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:36.240543    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" path="/var/lib/kubelet/pods/ea18e571-df1d-4711-aa0c-1c11ca6be4c3/volumes"
Nov 03 11:01:37.571770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.571725    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"]
Nov 03 11:01:37.572805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.572778    5393 cpu_manager.go:414] "containerMap: removing container" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy"
Nov 03 11:01:37.572971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.572807    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy"
Nov 03 11:01:37.572971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.572826    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy"
Nov 03 11:01:37.572971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.572834    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy"
Nov 03 11:01:37.572971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.572905    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy"
Nov 03 11:01:37.572971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.572931    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy"
Nov 03 11:01:37.573249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.573112    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="da57fce0-41d2-4546-a567-7dcc3bdc9221" containerName="istio-proxy"
Nov 03 11:01:37.573249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.573127    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea18e571-df1d-4711-aa0c-1c11ca6be4c3" containerName="istio-proxy"
Nov 03 11:01:37.573249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.573150    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68f2cb2-4b7a-49d1-9066-bf6b2f504ca7" containerName="istio-proxy"
Nov 03 11:01:37.581952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.579312    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.598462 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.598332    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drh62\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-kube-api-access-drh62\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.599210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.599184    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-token\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.599429 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.599410    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istiod-ca-cert\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.599602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.599585    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-certs\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.599765 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.599748    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-envoy\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.599905 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.599889    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-podinfo\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.600134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.598851    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b76c62-openshift-default-dockercfg-4s2l6\""
Nov 03 11:01:37.600250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.600231    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-credential-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.600551 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.600528    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-data\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.600756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.600731    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.611960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.608171    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"]
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.719946    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-credential-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.720142    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-data\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.720212    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.720319    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drh62\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-kube-api-access-drh62\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.720363    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-token\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.720422    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istiod-ca-cert\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.720456    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-certs\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.720497    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-envoy\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.720946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.720536    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-podinfo\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.778942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.741673    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-credential-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.778942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.741960    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-data\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.778942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.742204    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-socket\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.778942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.742782    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istiod-ca-cert\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.778942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.743215    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-certs\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.778942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.767667    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-envoy\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.780586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.780556    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-podinfo\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.785942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.783443    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-token\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.794945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.792892    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drh62\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-kube-api-access-drh62\") pod \"e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") " pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:37.936375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:37.936298    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:38.398814 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:38.398785    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"]
Nov 03 11:01:38.400680 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:01:38.400649    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fa9ef1_cff1_4c95_bcea_b79ef3c5c1f9.slice/crio-a4dbcb09735593bc7e3bd30cf3bf3842139e8a2402030b166d8a38ab24ddf300 WatchSource:0}: Error finding container a4dbcb09735593bc7e3bd30cf3bf3842139e8a2402030b166d8a38ab24ddf300: Status 404 returned error can't find the container with id a4dbcb09735593bc7e3bd30cf3bf3842139e8a2402030b166d8a38ab24ddf300
Nov 03 11:01:38.403573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:38.403538    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:38.403696 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:38.403610    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:38.403696 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:38.403642    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:38.683086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:38.682988    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" event={"ID":"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9","Type":"ContainerStarted","Data":"4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b"}
Nov 03 11:01:38.683086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:38.683057    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" event={"ID":"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9","Type":"ContainerStarted","Data":"a4dbcb09735593bc7e3bd30cf3bf3842139e8a2402030b166d8a38ab24ddf300"}
Nov 03 11:01:38.704185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:38.704133    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podStartSLOduration=1.704115912 podStartE2EDuration="1.704115912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:38.701290933 +0000 UTC m=+1942.606145397" watchObservedRunningTime="2026-11-03 11:01:38.704115912 +0000 UTC m=+1942.608970354"
Nov 03 11:01:38.944024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:38.942378    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:39.944150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:39.944122    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:40.697869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:40.697838    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:40.698991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:40.698972    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:01:42.448045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.448012    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"]
Nov 03 11:01:42.452445 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.452427    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:42.458248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.458225    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b76c62\""
Nov 03 11:01:42.461505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.461485    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"]
Nov 03 11:01:42.525770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.525738    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"]
Nov 03 11:01:42.530073 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.530036    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:42.535184 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.535158    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"]
Nov 03 11:01:42.537065 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.537045    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b76c62-dockercfg-fpwn7\""
Nov 03 11:01:42.565302 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.565266    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqgn\" (UniqueName: \"kubernetes.io/projected/d923e57f-9487-4585-8fb9-c104d09b00aa-kube-api-access-dlqgn\") pod \"payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg\" (UID: \"d923e57f-9487-4585-8fb9-c104d09b00aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:42.565563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.565541    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923e57f-9487-4585-8fb9-c104d09b00aa-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg\" (UID: \"d923e57f-9487-4585-8fb9-c104d09b00aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:42.565668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.565652    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxct2\" (UniqueName: \"kubernetes.io/projected/d78d286e-868c-41b7-bafb-a016941cf7ab-kube-api-access-mxct2\") pod \"payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm\" (UID: \"d78d286e-868c-41b7-bafb-a016941cf7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:42.565779 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.565754    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78d286e-868c-41b7-bafb-a016941cf7ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm\" (UID: \"d78d286e-868c-41b7-bafb-a016941cf7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:42.672080 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.671960    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923e57f-9487-4585-8fb9-c104d09b00aa-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg\" (UID: \"d923e57f-9487-4585-8fb9-c104d09b00aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:42.672290 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.672088    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxct2\" (UniqueName: \"kubernetes.io/projected/d78d286e-868c-41b7-bafb-a016941cf7ab-kube-api-access-mxct2\") pod \"payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm\" (UID: \"d78d286e-868c-41b7-bafb-a016941cf7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:42.672290 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.672135    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78d286e-868c-41b7-bafb-a016941cf7ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm\" (UID: \"d78d286e-868c-41b7-bafb-a016941cf7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:42.672290 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.672259    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqgn\" (UniqueName: \"kubernetes.io/projected/d923e57f-9487-4585-8fb9-c104d09b00aa-kube-api-access-dlqgn\") pod \"payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg\" (UID: \"d923e57f-9487-4585-8fb9-c104d09b00aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:42.687945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.674141    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923e57f-9487-4585-8fb9-c104d09b00aa-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg\" (UID: \"d923e57f-9487-4585-8fb9-c104d09b00aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:42.712992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.700413    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78d286e-868c-41b7-bafb-a016941cf7ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm\" (UID: \"d78d286e-868c-41b7-bafb-a016941cf7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:42.733944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.732280    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqgn\" (UniqueName: \"kubernetes.io/projected/d923e57f-9487-4585-8fb9-c104d09b00aa-kube-api-access-dlqgn\") pod \"payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg\" (UID: \"d923e57f-9487-4585-8fb9-c104d09b00aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:42.737948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.734553    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxct2\" (UniqueName: \"kubernetes.io/projected/d78d286e-868c-41b7-bafb-a016941cf7ab-kube-api-access-mxct2\") pod \"payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm\" (UID: \"d78d286e-868c-41b7-bafb-a016941cf7ab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:42.791974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.791878    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:42.869115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:42.869081    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:43.631503 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:43.631474    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:01:43.633700 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:01:43.633491    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:01:43.646505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:43.645637    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:01:44.159188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:44.156157    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"]
Nov 03 11:01:45.019010 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.018963    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 11:01:45.175553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.165485    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:01:45.225647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.188885    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:01:45.226520 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.226474    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"]
Nov 03 11:01:45.226833 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.226807    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"]
Nov 03 11:01:45.246095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.246052    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:01:45.256388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.256343    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:01:45.287279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.287188    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"]
Nov 03 11:01:45.287635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.287602    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.318448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.318417    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"routing-overlay\""
Nov 03 11:01:45.319487 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.319462    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"kube-root-ca.crt\""
Nov 03 11:01:45.319729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.319713    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"openshift-service-ca.crt\""
Nov 03 11:01:45.319968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.319949    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-56687c\""
Nov 03 11:01:45.335448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.335413    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"agc-cred-5d8152330398\""
Nov 03 11:01:45.335855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.335833    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"payload-processing-external-model-e2e--885106fd-dockercfg-w2xvw\""
Nov 03 11:01:45.535535 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.535501    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.535953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.535908    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.536143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.536126    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.536367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.536350    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78tn\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-kube-api-access-z78tn\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.560014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.559907    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm" event={"ID":"d78d286e-868c-41b7-bafb-a016941cf7ab","Type":"ContainerStarted","Data":"cc2a6fbdff38c948129fad9c59c63e52e9f3ef01f91840fd02b6b6fe764bafac"}
Nov 03 11:01:45.562649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.562621    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg" event={"ID":"d923e57f-9487-4585-8fb9-c104d09b00aa","Type":"ContainerStarted","Data":"6f0fd5a3d65f41edcb2fc956502c9b95a90162884182b06585fb51db250b0c05"}
Nov 03 11:01:45.639955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.639163    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.639955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.639246    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.639955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.639340    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78tn\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-kube-api-access-z78tn\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.639955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.639446    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.648940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.640353    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.648940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.641423    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.648940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.644049    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.676212 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.675547    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78tn\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-kube-api-access-z78tn\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:45.810298 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.809993    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 11:01:45.810566 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.810544    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w" podUID="1b69c7ad-8e31-42c8-922d-b9dab996e617" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/health\": context deadline exceeded"
Nov 03 11:01:45.900683 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:45.899513    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:46.541947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:46.538812    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"]
Nov 03 11:01:46.550601 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:01:46.550563    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed5d139_d3c3_4739_ad7a_9be8852efe9d.slice/crio-3907cb9f14764f014f131bcd860319c5f233ac5fc844424621c01d6a9b7c1d52 WatchSource:0}: Error finding container 3907cb9f14764f014f131bcd860319c5f233ac5fc844424621c01d6a9b7c1d52: Status 404 returned error can't find the container with id 3907cb9f14764f014f131bcd860319c5f233ac5fc844424621c01d6a9b7c1d52
Nov 03 11:01:46.629180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:46.615449    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm" event={"ID":"d78d286e-868c-41b7-bafb-a016941cf7ab","Type":"ContainerStarted","Data":"4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673"}
Nov 03 11:01:46.629180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:46.615770    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:46.629180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:46.618953    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d" event={"ID":"eed5d139-d3c3-4739-ad7a-9be8852efe9d","Type":"ContainerStarted","Data":"3907cb9f14764f014f131bcd860319c5f233ac5fc844424621c01d6a9b7c1d52"}
Nov 03 11:01:46.651767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:46.651732    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg" event={"ID":"d923e57f-9487-4585-8fb9-c104d09b00aa","Type":"ContainerStarted","Data":"1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154"}
Nov 03 11:01:46.653294 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:46.653270    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:46.660959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:46.660545    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm" podStartSLOduration=4.660525896 podStartE2EDuration="4.660525896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:46.654312659 +0000 UTC m=+1950.559167101" watchObservedRunningTime="2026-11-03 11:01:46.660525896 +0000 UTC m=+1950.565380337"
Nov 03 11:01:46.675519 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:46.675468    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg" podStartSLOduration=4.675454627 podStartE2EDuration="4.675454627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:46.67124315 +0000 UTC m=+1950.576097586" watchObservedRunningTime="2026-11-03 11:01:46.675454627 +0000 UTC m=+1950.580309068"
Nov 03 11:01:47.720945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:47.720421    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d" event={"ID":"eed5d139-d3c3-4739-ad7a-9be8852efe9d","Type":"ContainerStarted","Data":"7f3acded20c8ead3ee39737ab491b04a0d117c0b1f631cf940aa739037ae819d"}
Nov 03 11:01:47.720945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:47.720489    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:47.754359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:47.754307    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d" podStartSLOduration=3.75428181 podStartE2EDuration="3.75428181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:01:47.751584674 +0000 UTC m=+1951.656439116" watchObservedRunningTime="2026-11-03 11:01:47.75428181 +0000 UTC m=+1951.659136306"
Nov 03 11:01:48.204940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.196455    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"]
Nov 03 11:01:48.256945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.255716    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.262945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.259979    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:01:48.262945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.260658    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:01:48.437947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.437648    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"]
Nov 03 11:01:48.468003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.463345    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.468003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.463395    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhmpf\" (UniqueName: \"kubernetes.io/projected/b08cca23-1a73-4fbe-9349-4a26b53fad91-kube-api-access-nhmpf\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.468003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.463436    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.573782 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.573748    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.574101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.574076    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhmpf\" (UniqueName: \"kubernetes.io/projected/b08cca23-1a73-4fbe-9349-4a26b53fad91-kube-api-access-nhmpf\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.574205 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.574194    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.576509 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.576489    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.637219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.637100    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.644952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.644886    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhmpf\" (UniqueName: \"kubernetes.io/projected/b08cca23-1a73-4fbe-9349-4a26b53fad91-kube-api-access-nhmpf\") pod \"maas-api-revoke-sub-454e3372-4jrvk\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:48.776125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:48.776035    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:49.349108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:49.349077    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"]
Nov 03 11:01:49.349849 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:01:49.349821    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08cca23_1a73_4fbe_9349_4a26b53fad91.slice/crio-cd8e98bcf3890805430b6573bf119ee216a420aef1b8a489acbd5eefb0550248 WatchSource:0}: Error finding container cd8e98bcf3890805430b6573bf119ee216a420aef1b8a489acbd5eefb0550248: Status 404 returned error can't find the container with id cd8e98bcf3890805430b6573bf119ee216a420aef1b8a489acbd5eefb0550248
Nov 03 11:01:49.782446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:49.782404    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk" event={"ID":"b08cca23-1a73-4fbe-9349-4a26b53fad91","Type":"ContainerStarted","Data":"cd8e98bcf3890805430b6573bf119ee216a420aef1b8a489acbd5eefb0550248"}
Nov 03 11:01:50.841473 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:50.841438    5393 generic.go:356] "Generic (PLEG): container finished" podID="b08cca23-1a73-4fbe-9349-4a26b53fad91" containerID="2f68401164b7a23ac52b562a62876bbd141201358a41df4b37af80c1187b2e79" exitCode=0
Nov 03 11:01:50.842052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:50.841477    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk" event={"ID":"b08cca23-1a73-4fbe-9349-4a26b53fad91","Type":"ContainerDied","Data":"2f68401164b7a23ac52b562a62876bbd141201358a41df4b37af80c1187b2e79"}
Nov 03 11:01:52.063102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.052048    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:52.166079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.165994    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-xks-ca\") pod \"b08cca23-1a73-4fbe-9349-4a26b53fad91\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") "
Nov 03 11:01:52.166445 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.166419    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-openshift-ca\") pod \"b08cca23-1a73-4fbe-9349-4a26b53fad91\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") "
Nov 03 11:01:52.166756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.166734    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08cca23-1a73-4fbe-9349-4a26b53fad91-kube-api-access-nhmpf\" (UniqueName: \"kubernetes.io/projected/b08cca23-1a73-4fbe-9349-4a26b53fad91-kube-api-access-nhmpf\") pod \"b08cca23-1a73-4fbe-9349-4a26b53fad91\" (UID: \"b08cca23-1a73-4fbe-9349-4a26b53fad91\") "
Nov 03 11:01:52.176370 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.176336    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-openshift-ca" pod "b08cca23-1a73-4fbe-9349-4a26b53fad91" (UID: "b08cca23-1a73-4fbe-9349-4a26b53fad91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:01:52.178462 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.178432    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-xks-ca" pod "b08cca23-1a73-4fbe-9349-4a26b53fad91" (UID: "b08cca23-1a73-4fbe-9349-4a26b53fad91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:01:52.178705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.178677    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08cca23-1a73-4fbe-9349-4a26b53fad91-kube-api-access-nhmpf" pod "b08cca23-1a73-4fbe-9349-4a26b53fad91" (UID: "b08cca23-1a73-4fbe-9349-4a26b53fad91"). InnerVolumeSpecName "kube-api-access-nhmpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:01:52.270015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.269973    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:52.270015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.270005    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08cca23-1a73-4fbe-9349-4a26b53fad91-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:52.270015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.270019    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhmpf\" (UniqueName: \"kubernetes.io/projected/b08cca23-1a73-4fbe-9349-4a26b53fad91-kube-api-access-nhmpf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:01:52.860866 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.860828    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"
Nov 03 11:01:52.860866 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.860849    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk" event={"ID":"b08cca23-1a73-4fbe-9349-4a26b53fad91","Type":"ContainerDied","Data":"cd8e98bcf3890805430b6573bf119ee216a420aef1b8a489acbd5eefb0550248"}
Nov 03 11:01:52.861141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:52.860928    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd8e98bcf3890805430b6573bf119ee216a420aef1b8a489acbd5eefb0550248"
Nov 03 11:01:54.484152 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:54.484098    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:01:54.484693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:54.484187    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:01:55.484133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:55.484092    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:01:55.484477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:55.484182    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:01:55.897296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:55.897262    5393 generic.go:356] "Generic (PLEG): container finished" podID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerID="004134a50ed4b5e5d0e2b8d95468203ce976a3411605a9b0135306af0b8eeec6" exitCode=0
Nov 03 11:01:55.898366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:55.897307    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" event={"ID":"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5","Type":"ContainerDied","Data":"004134a50ed4b5e5d0e2b8d95468203ce976a3411605a9b0135306af0b8eeec6"}
Nov 03 11:01:55.898366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:55.897722    5393 scope.go:122] "RemoveContainer" containerID="004134a50ed4b5e5d0e2b8d95468203ce976a3411605a9b0135306af0b8eeec6"
Nov 03 11:01:55.899968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:55.899911    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:55.900045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:55.900028    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:55.900097 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:55.900082    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:56.915001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:56.914969    5393 generic.go:356] "Generic (PLEG): container finished" podID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerID="74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e" exitCode=0
Nov 03 11:01:56.915445 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:56.915047    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" event={"ID":"41a9215b-e2bd-4992-a6d6-97add590eb3b","Type":"ContainerDied","Data":"74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e"}
Nov 03 11:01:56.915543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:56.915520    5393 scope.go:122] "RemoveContainer" containerID="74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e"
Nov 03 11:01:56.917516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:56.917490    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:56.917634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:56.917557    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:56.917634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:56.917598    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:01:56.917634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:56.917618    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" event={"ID":"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5","Type":"ContainerStarted","Data":"133e978b3f1dc3013feb07c7fe9656964f1bea2035660f04b6824cb97750e860"}
Nov 03 11:01:57.450125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:57.450088    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:01:57.450354 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:57.450325    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 11:01:57.450417 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:57.450384    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 11:01:57.753009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:57.746138    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:01:57.962013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:57.961967    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" event={"ID":"41a9215b-e2bd-4992-a6d6-97add590eb3b","Type":"ContainerStarted","Data":"5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506"}
Nov 03 11:01:58.449563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.449501    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 11:01:58.449753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.449584    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 11:01:58.602941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.602767    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:01:58.604448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.603656    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 11:01:58.604448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.603800    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 11:01:58.639939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.638980    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"]
Nov 03 11:01:58.640196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.640165    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b08cca23-1a73-4fbe-9349-4a26b53fad91" containerName="revoke-keys"
Nov 03 11:01:58.640196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.640190    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08cca23-1a73-4fbe-9349-4a26b53fad91" containerName="revoke-keys"
Nov 03 11:01:58.640385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.640367    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08cca23-1a73-4fbe-9349-4a26b53fad91" containerName="revoke-keys"
Nov 03 11:01:58.743685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.743604    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"]
Nov 03 11:01:58.744070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.744049    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.748710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.748686    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:01:58.749051 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.749035    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:01:58.750465 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.749789    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:01:58.752128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.751264    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:01:58.768536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.768504    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8dq\" (UniqueName: \"kubernetes.io/projected/07f8c0d1-345b-427e-9296-888f9491a2c7-kube-api-access-tn8dq\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.768705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.768665    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.768834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.768711    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.870619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.870575    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.870813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.870655    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.871419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.871388    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8dq\" (UniqueName: \"kubernetes.io/projected/07f8c0d1-345b-427e-9296-888f9491a2c7-kube-api-access-tn8dq\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.871575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.871446    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.873813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.873791    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:58.880355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:58.880338    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8dq\" (UniqueName: \"kubernetes.io/projected/07f8c0d1-345b-427e-9296-888f9491a2c7-kube-api-access-tn8dq\") pod \"maas-api-revoke-sub-82e2a7fe-6kvbw\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:59.070469 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:59.070387    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:01:59.449497 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:59.449468    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 11:01:59.449630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:59.449529    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 11:01:59.451134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:59.451105    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"]
Nov 03 11:01:59.452469 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:01:59.452443    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f8c0d1_345b_427e_9296_888f9491a2c7.slice/crio-94e95aa1ac380335cf404d1c5cd54df827bca32e0cb7d2e828ee2a211f58cb2c WatchSource:0}: Error finding container 94e95aa1ac380335cf404d1c5cd54df827bca32e0cb7d2e828ee2a211f58cb2c: Status 404 returned error can't find the container with id 94e95aa1ac380335cf404d1c5cd54df827bca32e0cb7d2e828ee2a211f58cb2c
Nov 03 11:01:59.612952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:59.610448    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 11:01:59.612952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:01:59.610526    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 11:02:00.046195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:00.046162    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw" event={"ID":"07f8c0d1-345b-427e-9296-888f9491a2c7","Type":"ContainerStarted","Data":"94e95aa1ac380335cf404d1c5cd54df827bca32e0cb7d2e828ee2a211f58cb2c"}
Nov 03 11:02:00.450738 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:00.450701    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 11:02:00.451206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:00.450767    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 11:02:00.619945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:00.619055    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 11:02:00.619945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:00.619120    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 11:02:01.089945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:01.088905    5393 generic.go:356] "Generic (PLEG): container finished" podID="07f8c0d1-345b-427e-9296-888f9491a2c7" containerID="d77632c9dacde739fc9e10e8a24743c6b38bb9c39692b8e6ec950c0772ec1fa5" exitCode=0
Nov 03 11:02:01.089945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:01.089596    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw" event={"ID":"07f8c0d1-345b-427e-9296-888f9491a2c7","Type":"ContainerDied","Data":"d77632c9dacde739fc9e10e8a24743c6b38bb9c39692b8e6ec950c0772ec1fa5"}
Nov 03 11:02:01.478444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:01.478397    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 11:02:01.479063 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:01.479030    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 11:02:01.600420 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:01.600383    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 11:02:01.601424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:01.601392    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 11:02:02.428753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.428675    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:02:02.447505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.445709    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f8c0d1-345b-427e-9296-888f9491a2c7-kube-api-access-tn8dq\" (UniqueName: \"kubernetes.io/projected/07f8c0d1-345b-427e-9296-888f9491a2c7-kube-api-access-tn8dq\") pod \"07f8c0d1-345b-427e-9296-888f9491a2c7\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") "
Nov 03 11:02:02.447505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.445847    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-openshift-ca\") pod \"07f8c0d1-345b-427e-9296-888f9491a2c7\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") "
Nov 03 11:02:02.447505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.445951    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-xks-ca\") pod \"07f8c0d1-345b-427e-9296-888f9491a2c7\" (UID: \"07f8c0d1-345b-427e-9296-888f9491a2c7\") "
Nov 03 11:02:02.447505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.447311    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-openshift-ca" pod "07f8c0d1-345b-427e-9296-888f9491a2c7" (UID: "07f8c0d1-345b-427e-9296-888f9491a2c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:02.449891 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.449862    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 11:02:02.450076 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.449935    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 11:02:02.450991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.450687    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f8c0d1-345b-427e-9296-888f9491a2c7-kube-api-access-tn8dq" pod "07f8c0d1-345b-427e-9296-888f9491a2c7" (UID: "07f8c0d1-345b-427e-9296-888f9491a2c7"). InnerVolumeSpecName "kube-api-access-tn8dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:02.450991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.450792    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-xks-ca" pod "07f8c0d1-345b-427e-9296-888f9491a2c7" (UID: "07f8c0d1-345b-427e-9296-888f9491a2c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:02:02.548292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.548245    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn8dq\" (UniqueName: \"kubernetes.io/projected/07f8c0d1-345b-427e-9296-888f9491a2c7-kube-api-access-tn8dq\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:02.548292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.548289    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:02.548292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.548301    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f8c0d1-345b-427e-9296-888f9491a2c7-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:02.602866 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.602408    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 11:02:02.602866 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:02.602471    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 11:02:03.132306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:03.132280    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"
Nov 03 11:02:03.132538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:03.132348    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw" event={"ID":"07f8c0d1-345b-427e-9296-888f9491a2c7","Type":"ContainerDied","Data":"94e95aa1ac380335cf404d1c5cd54df827bca32e0cb7d2e828ee2a211f58cb2c"}
Nov 03 11:02:03.132538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:03.132419    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e95aa1ac380335cf404d1c5cd54df827bca32e0cb7d2e828ee2a211f58cb2c"
Nov 03 11:02:03.455841 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:03.455750    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:02:03.456261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:03.456232    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:02:03.457025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:03.457007    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:02:03.599393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:03.599354    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 11:02:03.599812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:03.599420    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 11:02:04.604979 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:04.604946    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:02:04.605522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:04.605495    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:02:04.606163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:04.606144    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:02:10.418069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:10.411569    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"]
Nov 03 11:02:13.190539 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:02:13.190506    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:02:18.398500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:18.398455    5393 scope.go:122] "RemoveContainer" containerID="6bff190cb85973accb1e3e2e6bffff99991080c08429932b89e1b054b1458c3c"
Nov 03 11:02:25.451273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.451219    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf" podUID="3f7a8de3-279c-47bc-8086-834be967a412" containerName="main" containerID="cri-o://263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348" gracePeriod=45
Nov 03 11:02:25.809938 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.808235    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 11:02:25.862572 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.862533    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-dshm\") pod \"3f7a8de3-279c-47bc-8086-834be967a412\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") "
Nov 03 11:02:25.862738 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.862624    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-model-cache\") pod \"3f7a8de3-279c-47bc-8086-834be967a412\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") "
Nov 03 11:02:25.862876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.862855    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-home\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-home\") pod \"3f7a8de3-279c-47bc-8086-834be967a412\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") "
Nov 03 11:02:25.862989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.862911    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f7a8de3-279c-47bc-8086-834be967a412-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7a8de3-279c-47bc-8086-834be967a412-tls-certs\") pod \"3f7a8de3-279c-47bc-8086-834be967a412\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") "
Nov 03 11:02:25.863078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.863058    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7a8de3-279c-47bc-8086-834be967a412-kube-api-access-9mhk7\" (UniqueName: \"kubernetes.io/projected/3f7a8de3-279c-47bc-8086-834be967a412-kube-api-access-9mhk7\") pod \"3f7a8de3-279c-47bc-8086-834be967a412\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") "
Nov 03 11:02:25.863220 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.863201    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-tmp-dir\") pod \"3f7a8de3-279c-47bc-8086-834be967a412\" (UID: \"3f7a8de3-279c-47bc-8086-834be967a412\") "
Nov 03 11:02:25.864180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.864151    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-tmp-dir" pod "3f7a8de3-279c-47bc-8086-834be967a412" (UID: "3f7a8de3-279c-47bc-8086-834be967a412"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:25.864385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.864364    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-model-cache" pod "3f7a8de3-279c-47bc-8086-834be967a412" (UID: "3f7a8de3-279c-47bc-8086-834be967a412"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:25.864564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.864542    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-home" pod "3f7a8de3-279c-47bc-8086-834be967a412" (UID: "3f7a8de3-279c-47bc-8086-834be967a412"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:25.869533 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.869493    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-dshm" pod "3f7a8de3-279c-47bc-8086-834be967a412" (UID: "3f7a8de3-279c-47bc-8086-834be967a412"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:25.870017 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.869994    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7a8de3-279c-47bc-8086-834be967a412-tls-certs" pod "3f7a8de3-279c-47bc-8086-834be967a412" (UID: "3f7a8de3-279c-47bc-8086-834be967a412"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:02:25.870105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.870029    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7a8de3-279c-47bc-8086-834be967a412-kube-api-access-9mhk7" pod "3f7a8de3-279c-47bc-8086-834be967a412" (UID: "3f7a8de3-279c-47bc-8086-834be967a412"). InnerVolumeSpecName "kube-api-access-9mhk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:25.983006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.972863    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:25.983006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.972897    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7a8de3-279c-47bc-8086-834be967a412-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:25.983006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.972936    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mhk7\" (UniqueName: \"kubernetes.io/projected/3f7a8de3-279c-47bc-8086-834be967a412-kube-api-access-9mhk7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:25.983006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.972954    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:25.983006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.972967    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:25.983006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:25.972979    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7a8de3-279c-47bc-8086-834be967a412-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:26.451107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.451068    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"]
Nov 03 11:02:26.451316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.451291    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy" containerID="cri-o://583800af586b3b77b1d3f776b03ad864b7482a41194ce006cabb13055211c103" gracePeriod=30
Nov 03 11:02:26.499029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.498897    5393 generic.go:356] "Generic (PLEG): container finished" podID="3f7a8de3-279c-47bc-8086-834be967a412" containerID="263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348" exitCode=0
Nov 03 11:02:26.499029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.498987    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf" event={"ID":"3f7a8de3-279c-47bc-8086-834be967a412","Type":"ContainerDied","Data":"263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348"}
Nov 03 11:02:26.499029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.499024    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf" event={"ID":"3f7a8de3-279c-47bc-8086-834be967a412","Type":"ContainerDied","Data":"901f9f9c72d953febaeba275f329013c852aee7f2a2deb0fc50fc6cfd31bff3a"}
Nov 03 11:02:26.499271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.499045    5393 scope.go:122] "RemoveContainer" containerID="263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348"
Nov 03 11:02:26.499271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.499203    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"
Nov 03 11:02:26.517092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.517063    5393 scope.go:122] "RemoveContainer" containerID="263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348"
Nov 03 11:02:26.517449 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:02:26.517423    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348\": container with ID starting with 263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348 not found: ID does not exist" containerID="263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348"
Nov 03 11:02:26.517546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.517460    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348"} err="failed to get container status \"263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348\": rpc error: code = NotFound desc = could not find container \"263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348\": container with ID starting with 263d6bcb28b349cbe5f5f20ca2d7de53c1270ecac642bcf987c2046dfd2f4348 not found: ID does not exist"
Nov 03 11:02:26.519560 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.519532    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"]
Nov 03 11:02:26.525324 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.525295    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf"]
Nov 03 11:02:26.733531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.733485    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-47a8a4/auth-test-model-47a8a4-kserve-57ffdd7b85-mhkjf" podUID="3f7a8de3-279c-47bc-8086-834be967a412" containerName="main" probeResult="failure" output="Get \"https://10.134.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:02:26.763256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.763028    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:02:26.763256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:26.763111    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:02:28.234595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:28.234555    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7a8de3-279c-47bc-8086-834be967a412" path="/var/lib/kubelet/pods/3f7a8de3-279c-47bc-8086-834be967a412/volumes"
Nov 03 11:02:30.609845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:30.609805    5393 generic.go:356] "Generic (PLEG): container finished" podID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerID="4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b" exitCode=0
Nov 03 11:02:30.610355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:30.609883    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" event={"ID":"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9","Type":"ContainerDied","Data":"4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b"}
Nov 03 11:02:30.610355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:30.610334    5393 scope.go:122] "RemoveContainer" containerID="4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b"
Nov 03 11:02:30.612245 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:30.612209    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:30.612356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:30.612282    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:30.612356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:30.612316    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:02:31.689956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:31.687093    5393 generic.go:356] "Generic (PLEG): container finished" podID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerID="583800af586b3b77b1d3f776b03ad864b7482a41194ce006cabb13055211c103" exitCode=0
Nov 03 11:02:31.689956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:31.687344    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" event={"ID":"44ef1d3a-2ef8-40c8-818e-b35ecc26150b","Type":"ContainerDied","Data":"583800af586b3b77b1d3f776b03ad864b7482a41194ce006cabb13055211c103"}
Nov 03 11:02:31.701947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:31.691224    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" event={"ID":"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9","Type":"ContainerStarted","Data":"1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d"}
Nov 03 11:02:31.938262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:31.938218    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:02:31.938564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:31.938525    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 11:02:31.938823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:31.938796    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 11:02:31.941545 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:31.941521    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 11:02:32.026511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.026413    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-token\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-token\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.026692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.026528    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-envoy\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.026692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.026579    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-certs\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.026809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.026695    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-socket\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.026809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.026797    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-kube-api-access-cmjkp\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-kube-api-access-cmjkp\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.026977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.026852    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-credential-socket\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.026977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.026882    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-data\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.027099 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.027000    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-podinfo\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.027160 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.027136    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istiod-ca-cert\") pod \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\" (UID: \"44ef1d3a-2ef8-40c8-818e-b35ecc26150b\") "
Nov 03 11:02:32.028072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.028039    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istiod-ca-cert" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:32.028748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.028720    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-data" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:32.029022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.028967    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-credential-socket" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:32.029194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.029169    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-certs" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:32.029793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.029764    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-socket" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:32.031858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.031811    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-token" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:32.032315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.032272    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-podinfo" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:02:32.032428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.032389    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-kube-api-access-cmjkp" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "kube-api-access-cmjkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:32.034420 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.034346    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-envoy" pod "44ef1d3a-2ef8-40c8-818e-b35ecc26150b" (UID: "44ef1d3a-2ef8-40c8-818e-b35ecc26150b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:32.129110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129058    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.129110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129091    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.129110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129104    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.129110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129118    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.129456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129131    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.129456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129146    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.129456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129161    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmjkp\" (UniqueName: \"kubernetes.io/projected/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-kube-api-access-cmjkp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.129456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129175    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.129456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.129188    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ef1d3a-2ef8-40c8-818e-b35ecc26150b-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:32.360528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.360493    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"]
Nov 03 11:02:32.361249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361228    5393 cpu_manager.go:414] "containerMap: removing container" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy"
Nov 03 11:02:32.361249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361249    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy"
Nov 03 11:02:32.361424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361287    5393 cpu_manager.go:414] "containerMap: removing container" podUID="07f8c0d1-345b-427e-9296-888f9491a2c7" containerName="revoke-keys"
Nov 03 11:02:32.361424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361295    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f8c0d1-345b-427e-9296-888f9491a2c7" containerName="revoke-keys"
Nov 03 11:02:32.361424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361314    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3f7a8de3-279c-47bc-8086-834be967a412" containerName="main"
Nov 03 11:02:32.361424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361323    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7a8de3-279c-47bc-8086-834be967a412" containerName="main"
Nov 03 11:02:32.361649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361536    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f8c0d1-345b-427e-9296-888f9491a2c7" containerName="revoke-keys"
Nov 03 11:02:32.361649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361554    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7a8de3-279c-47bc-8086-834be967a412" containerName="main"
Nov 03 11:02:32.361649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.361577    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" containerName="istio-proxy"
Nov 03 11:02:32.364715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.364695    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:32.371984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.371963    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9f53bb\""
Nov 03 11:02:32.374244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.374220    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"]
Nov 03 11:02:32.438506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.438470    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkf7\" (UniqueName: \"kubernetes.io/projected/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-kube-api-access-qpkf7\") pod \"payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g\" (UID: \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:32.438724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.438689    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g\" (UID: \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:32.540959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.539983    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkf7\" (UniqueName: \"kubernetes.io/projected/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-kube-api-access-qpkf7\") pod \"payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g\" (UID: \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:32.540959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.540175    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g\" (UID: \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:32.544959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.541266    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g\" (UID: \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:32.605006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.604904    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkf7\" (UniqueName: \"kubernetes.io/projected/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-kube-api-access-qpkf7\") pod \"payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g\" (UID: \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:32.719881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.718486    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:32.739943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.733244    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"
Nov 03 11:02:32.739943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.733296    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx" event={"ID":"44ef1d3a-2ef8-40c8-818e-b35ecc26150b","Type":"ContainerDied","Data":"ee1621285be07a73a2dc0cc4354b5b739cfadde1e0a166bff2c8ece9550acf24"}
Nov 03 11:02:32.739943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.733354    5393 scope.go:122] "RemoveContainer" containerID="583800af586b3b77b1d3f776b03ad864b7482a41194ce006cabb13055211c103"
Nov 03 11:02:32.826046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.825968    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"]
Nov 03 11:02:32.850685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.850331    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d7gsx"]
Nov 03 11:02:32.938086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.938054    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 11:02:32.938263 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:32.938102    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 11:02:33.172127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.172100    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"]
Nov 03 11:02:33.174196 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:02:33.174172    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbf5e5c_cfe2_412c_941b_a25b9d12b9e6.slice/crio-2e4250bd0dc0b45732b2ec0e9116e47ca511ccc39e4ee95b0c4a4b899e69653b WatchSource:0}: Error finding container 2e4250bd0dc0b45732b2ec0e9116e47ca511ccc39e4ee95b0c4a4b899e69653b: Status 404 returned error can't find the container with id 2e4250bd0dc0b45732b2ec0e9116e47ca511ccc39e4ee95b0c4a4b899e69653b
Nov 03 11:02:33.743314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.743262    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g" event={"ID":"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6","Type":"ContainerStarted","Data":"7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b"}
Nov 03 11:02:33.743800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.743348    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g" event={"ID":"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6","Type":"ContainerStarted","Data":"2e4250bd0dc0b45732b2ec0e9116e47ca511ccc39e4ee95b0c4a4b899e69653b"}
Nov 03 11:02:33.743906 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.743887    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:33.760562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.760491    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g" podStartSLOduration=1.760477384 podStartE2EDuration="1.760477384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:02:33.759192516 +0000 UTC m=+1997.664046959" watchObservedRunningTime="2026-11-03 11:02:33.760477384 +0000 UTC m=+1997.665331824"
Nov 03 11:02:33.840398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.840349    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"]
Nov 03 11:02:33.840620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.840595    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy" containerID="cri-o://3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d" gracePeriod=30
Nov 03 11:02:33.843912 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.843887    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"]
Nov 03 11:02:33.844137 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.844116    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy" containerID="cri-o://9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad" gracePeriod=30
Nov 03 11:02:33.847167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.847141    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"]
Nov 03 11:02:33.847306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.847285    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy" containerID="cri-o://4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb" gracePeriod=30
Nov 03 11:02:33.939102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.938804    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 11:02:33.939102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:33.938875    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 11:02:34.269974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:34.266076    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ef1d3a-2ef8-40c8-818e-b35ecc26150b" path="/var/lib/kubelet/pods/44ef1d3a-2ef8-40c8-818e-b35ecc26150b/volumes"
Nov 03 11:02:34.938825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:34.938784    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 11:02:34.939278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:34.938842    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 11:02:35.938439 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:35.938210    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 11:02:35.938439 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:35.938287    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 11:02:36.939547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:36.938067    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 11:02:36.939547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:36.938140    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 11:02:37.944258 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:37.944219    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:02:37.944723 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:37.944672    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:02:37.945510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:37.945491    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:02:39.227637 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.227608    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 11:02:39.269695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.269664    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 11:02:39.287426 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.287400    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 11:02:39.297214 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297181    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-data\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.297376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297282    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-certs\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.297376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297349    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istiod-ca-cert\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.297505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297392    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-socket\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.297505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297448    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-certs\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.297602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297506    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-kube-api-access-vvscj\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-kube-api-access-vvscj\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.297602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297538    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-credential-socket\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.297602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297584    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-socket\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.297748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297616    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-data\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.297748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.297659    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a50084-678c-42e3-b5be-bbc87cfd0574-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a50084-678c-42e3-b5be-bbc87cfd0574-istiod-ca-cert\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.298294 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.298119    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-certs" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.298428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.298311    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-credential-socket\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.298428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.298392    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-podinfo\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.298546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.298456    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-envoy\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.298546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.298485    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-socket\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.298546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.298535    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-token\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-token\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.298710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.298567    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istiod-ca-cert\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.299067 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.298902    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-kube-api-access-jcgtd\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-kube-api-access-jcgtd\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.299067 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299009    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istiod-ca-cert" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:39.299067 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299053    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-certs\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.299280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299124    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-podinfo\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.299280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299237    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-podinfo\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.299498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299345    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-data\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.299498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299413    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-envoy\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.299498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299442    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-token\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-token\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.299498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299448    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-socket" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.299712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299647    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-certs" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.299762 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.299710    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-credential-socket" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.300035 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.300014    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a50084-678c-42e3-b5be-bbc87cfd0574-istiod-ca-cert" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:39.300374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.300351    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.300467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.300378    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.300467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.300392    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.300467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.300407    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.300467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.300422    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.300467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.300435    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a50084-678c-42e3-b5be-bbc87cfd0574-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.300879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.300808    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-data" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.301280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.301257    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-socket" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.302308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.301466    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-certs" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.302308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.301636    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-data" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.302308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.301646    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-data" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.302308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.302026    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-socket" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.302308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.302154    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-credential-socket" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.304113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.304079    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istiod-ca-cert" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:02:39.304532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.304505    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-podinfo" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:02:39.305170 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.305118    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-envoy" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.305274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.305239    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-podinfo" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:02:39.306004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.305981    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-envoy" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.306168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.306011    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-token" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:39.306168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.306043    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-kube-api-access-vvscj" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "kube-api-access-vvscj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:39.306168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.306084    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-podinfo" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:02:39.307339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.307315    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-token" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:39.309578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.309549    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-kube-api-access-jcgtd" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "kube-api-access-jcgtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:39.401487 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.401413    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-kube-api-access-vd8g5\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-kube-api-access-vd8g5\") pod \"00a50084-678c-42e3-b5be-bbc87cfd0574\" (UID: \"00a50084-678c-42e3-b5be-bbc87cfd0574\") "
Nov 03 11:02:39.401487 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.401482    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-credential-socket\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.401671 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.401521    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-envoy\") pod \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\" (UID: \"1b47fb10-dfcd-4463-bc70-889f926ce1d2\") "
Nov 03 11:02:39.401792 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.401773    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-token\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-token\") pod \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\" (UID: \"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0\") "
Nov 03 11:02:39.401837 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.401798    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-credential-socket" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.403088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403058    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvscj\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-kube-api-access-vvscj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403093    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403109    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403122    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403136    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403149    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403163    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403176    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403190    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403204    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcgtd\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-kube-api-access-jcgtd\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403218    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403232    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403245    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403259    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403272    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403285    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a50084-678c-42e3-b5be-bbc87cfd0574-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403299    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.403859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403312    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.404155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403911    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-envoy" pod "1b47fb10-dfcd-4463-bc70-889f926ce1d2" (UID: "1b47fb10-dfcd-4463-bc70-889f926ce1d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:02:39.404155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.403911    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-token" pod "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" (UID: "e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:39.404155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.404101    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-kube-api-access-vd8g5" pod "00a50084-678c-42e3-b5be-bbc87cfd0574" (UID: "00a50084-678c-42e3-b5be-bbc87cfd0574"). InnerVolumeSpecName "kube-api-access-vd8g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:02:39.504558 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.504523    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8g5\" (UniqueName: \"kubernetes.io/projected/00a50084-678c-42e3-b5be-bbc87cfd0574-kube-api-access-vd8g5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.504558 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.504555    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b47fb10-dfcd-4463-bc70-889f926ce1d2-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.504558 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.504566    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:02:39.806041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.805937    5393 generic.go:356] "Generic (PLEG): container finished" podID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerID="4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb" exitCode=0
Nov 03 11:02:39.806041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.805981    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" event={"ID":"1b47fb10-dfcd-4463-bc70-889f926ce1d2","Type":"ContainerDied","Data":"4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb"}
Nov 03 11:02:39.806041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.806001    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"
Nov 03 11:02:39.806041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.806040    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7" event={"ID":"1b47fb10-dfcd-4463-bc70-889f926ce1d2","Type":"ContainerDied","Data":"cb837bdecdaaf2b35e15b9adc6f70571f348b9f264ad089e5d32bccc75f84fbf"}
Nov 03 11:02:39.806350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.806073    5393 scope.go:122] "RemoveContainer" containerID="4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb"
Nov 03 11:02:39.807754 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.807729    5393 generic.go:356] "Generic (PLEG): container finished" podID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerID="9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad" exitCode=0
Nov 03 11:02:39.807876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.807795    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"
Nov 03 11:02:39.807876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.807825    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" event={"ID":"00a50084-678c-42e3-b5be-bbc87cfd0574","Type":"ContainerDied","Data":"9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad"}
Nov 03 11:02:39.808087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.807896    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4" event={"ID":"00a50084-678c-42e3-b5be-bbc87cfd0574","Type":"ContainerDied","Data":"b4f52989e115d412c2ce01717b3435e84c8be23bff9b1e01e9ea212e0b5022ad"}
Nov 03 11:02:39.813865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.813839    5393 generic.go:356] "Generic (PLEG): container finished" podID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerID="3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d" exitCode=0
Nov 03 11:02:39.814716 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.814694    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"
Nov 03 11:02:39.815031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.815005    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" event={"ID":"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0","Type":"ContainerDied","Data":"3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d"}
Nov 03 11:02:39.815147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.815060    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7" event={"ID":"e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0","Type":"ContainerDied","Data":"397df1d0192fa33f8dfee186fbb7950ca96821f7a00b554b4740d8504d012eb7"}
Nov 03 11:02:39.819038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.818945    5393 scope.go:122] "RemoveContainer" containerID="4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb"
Nov 03 11:02:39.819383 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:02:39.819354    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb\": container with ID starting with 4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb not found: ID does not exist" containerID="4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb"
Nov 03 11:02:39.819497 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.819390    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb"} err="failed to get container status \"4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb\": rpc error: code = NotFound desc = could not find container \"4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb\": container with ID starting with 4ceb5f5a7690bc9ed57eba742d51c1ac07c3854fb651b7fc59dec9e09802b4eb not found: ID does not exist"
Nov 03 11:02:39.819497 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.819414    5393 scope.go:122] "RemoveContainer" containerID="9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad"
Nov 03 11:02:39.829815 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.829787    5393 scope.go:122] "RemoveContainer" containerID="9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad"
Nov 03 11:02:39.830106 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:02:39.830085    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad\": container with ID starting with 9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad not found: ID does not exist" containerID="9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad"
Nov 03 11:02:39.830179 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.830113    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad"} err="failed to get container status \"9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad\": rpc error: code = NotFound desc = could not find container \"9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad\": container with ID starting with 9ec2e8cd9e62223983b578c63e45e6f1b6e9cd294f70489e2c59bcbe1ae842ad not found: ID does not exist"
Nov 03 11:02:39.830179 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.830129    5393 scope.go:122] "RemoveContainer" containerID="3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d"
Nov 03 11:02:39.836424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.836399    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"]
Nov 03 11:02:39.840403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.840377    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-dpls7"]
Nov 03 11:02:39.841816 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.841798    5393 scope.go:122] "RemoveContainer" containerID="3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d"
Nov 03 11:02:39.842164 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:02:39.842139    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d\": container with ID starting with 3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d not found: ID does not exist" containerID="3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d"
Nov 03 11:02:39.842235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.842175    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d"} err="failed to get container status \"3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d\": rpc error: code = NotFound desc = could not find container \"3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d\": container with ID starting with 3b212db4fe16afe3393bd8c346c59cfaa1b03a56374d1bec514117353bf8343d not found: ID does not exist"
Nov 03 11:02:39.849168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.849133    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"]
Nov 03 11:02:39.852683 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.852661    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-wknz4"]
Nov 03 11:02:39.864179 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.864157    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"]
Nov 03 11:02:39.870221 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:39.870198    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-6csq7"]
Nov 03 11:02:40.239928 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:40.239890    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" path="/var/lib/kubelet/pods/00a50084-678c-42e3-b5be-bbc87cfd0574/volumes"
Nov 03 11:02:40.240593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:40.240572    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" path="/var/lib/kubelet/pods/1b47fb10-dfcd-4463-bc70-889f926ce1d2/volumes"
Nov 03 11:02:40.241103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:40.241083    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" path="/var/lib/kubelet/pods/e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0/volumes"
Nov 03 11:02:43.221761 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:02:43.221710    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:02:44.746495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.746447    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"]
Nov 03 11:02:44.747324 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747304    5393 cpu_manager.go:414] "containerMap: removing container" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy"
Nov 03 11:02:44.747382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747326    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy"
Nov 03 11:02:44.747382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747337    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy"
Nov 03 11:02:44.747382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747341    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy"
Nov 03 11:02:44.747382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747365    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy"
Nov 03 11:02:44.747382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747370    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy"
Nov 03 11:02:44.747534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747460    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a50084-678c-42e3-b5be-bbc87cfd0574" containerName="istio-proxy"
Nov 03 11:02:44.747534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747468    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b47fb10-dfcd-4463-bc70-889f926ce1d2" containerName="istio-proxy"
Nov 03 11:02:44.747534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.747480    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6de65f4-8d63-4a2b-b3e5-dc462e3f84d0" containerName="istio-proxy"
Nov 03 11:02:44.750839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.750821    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.757957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.757929    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b76c62\""
Nov 03 11:02:44.758172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.758150    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"routing-overlay\""
Nov 03 11:02:44.758270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.758238    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"kube-root-ca.crt\""
Nov 03 11:02:44.759057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.758717    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"openshift-service-ca.crt\""
Nov 03 11:02:44.759057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.758791    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"agc-cred-ff8b9f4705b7\""
Nov 03 11:02:44.759057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.758938    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"payload-processing-external-model-e2e--0d6b7192-dockercfg-97m2g\""
Nov 03 11:02:44.764437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.764400    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"]
Nov 03 11:02:44.786167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.786133    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:02:44.886667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.886628    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.886872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.886699    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.887058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.887035    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6g5\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-kube-api-access-7b6g5\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.887162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.887146    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.994758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.990827    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.994758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.990906    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.994758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.991157    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6g5\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-kube-api-access-7b6g5\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.994758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.991248    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.994758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.992468    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:44.994758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:44.993161    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:45.026992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:45.019431    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:45.042176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:45.036454    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6g5\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-kube-api-access-7b6g5\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:45.106959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:45.103669    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:45.581267 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:45.581235    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"]
Nov 03 11:02:45.582167 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:02:45.582138    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cadeaf7_bdeb_442e_b6d8_db7b96118ea1.slice/crio-1fc6277d2e23ece9287ddcface6d831ccc92a4ea5e6c29b914f92e65357d8ca2 WatchSource:0}: Error finding container 1fc6277d2e23ece9287ddcface6d831ccc92a4ea5e6c29b914f92e65357d8ca2: Status 404 returned error can't find the container with id 1fc6277d2e23ece9287ddcface6d831ccc92a4ea5e6c29b914f92e65357d8ca2
Nov 03 11:02:45.867937 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:45.867834    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b" event={"ID":"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1","Type":"ContainerStarted","Data":"928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7"}
Nov 03 11:02:45.867937 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:45.867901    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b" event={"ID":"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1","Type":"ContainerStarted","Data":"1fc6277d2e23ece9287ddcface6d831ccc92a4ea5e6c29b914f92e65357d8ca2"}
Nov 03 11:02:45.868349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:45.867971    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:45.885584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:45.885531    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b" podStartSLOduration=1.8855147840000002 podStartE2EDuration="1.885514784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:02:45.883630592 +0000 UTC m=+2009.788485034" watchObservedRunningTime="2026-11-03 11:02:45.885514784 +0000 UTC m=+2009.790369224"
Nov 03 11:02:47.871113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:47.871077    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"]
Nov 03 11:02:47.875499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:47.875472    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:47.881667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:47.881645    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9f53bb-serving-cert\""
Nov 03 11:02:47.890086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:47.890065    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"]
Nov 03 11:02:47.977249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:47.977215    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:47.977431 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:47.977345    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829zg\" (UniqueName: \"kubernetes.io/projected/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-kube-api-access-829zg\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:47.977528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:47.977430    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:47.977528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:47.977500    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-metrics-certs\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.118079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.118014    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.118460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.118439    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829zg\" (UniqueName: \"kubernetes.io/projected/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-kube-api-access-829zg\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.118605 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.118590    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.118744 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.118729    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-metrics-certs\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.128266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.122891    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-metrics-certs\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.128266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.123573    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.133422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.132384    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.136724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.136701    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-829zg\" (UniqueName: \"kubernetes.io/projected/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-kube-api-access-829zg\") pod \"maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.194171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.194135    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:48.482153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.481099    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"]
Nov 03 11:02:48.493953 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:02:48.487382    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dbc1b4_1267_40f9_90d5_f8ef3faea21f.slice/crio-96df586c1e4dac2f140e2e44803c20827c933944fe0a3df8325f1e3fcc3d57f5 WatchSource:0}: Error finding container 96df586c1e4dac2f140e2e44803c20827c933944fe0a3df8325f1e3fcc3d57f5: Status 404 returned error can't find the container with id 96df586c1e4dac2f140e2e44803c20827c933944fe0a3df8325f1e3fcc3d57f5
Nov 03 11:02:48.525954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.513851    5393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:02:48.893117 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:48.893081    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h" event={"ID":"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f","Type":"ContainerStarted","Data":"96df586c1e4dac2f140e2e44803c20827c933944fe0a3df8325f1e3fcc3d57f5"}
Nov 03 11:02:49.902164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:49.902129    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h" event={"ID":"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f","Type":"ContainerStarted","Data":"a2200e39162bdf41e4e5de9081195488f6aa7bc77ca4157115d6de70f80ab36d"}
Nov 03 11:02:49.902643 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:49.902225    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:49.920578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:49.920527    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h" podStartSLOduration=2.393203519 podStartE2EDuration="2.920511742s" totalImagesPullingTime="527.308223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:02:49.918335097 +0000 UTC m=+2013.823189552" watchObservedRunningTime="2026-11-03 11:02:49.920511742 +0000 UTC m=+2013.825366183"
Nov 03 11:02:55.915807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:55.915777    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:02:56.879127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:56.879045    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:02:58.558708 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:58.558677    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-3410cd"]
Nov 03 11:02:58.562122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:58.562103    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:02:58.566164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:58.566139    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5zm6k\""
Nov 03 11:02:58.589397 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:58.589363    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-3410cd"]
Nov 03 11:02:58.677486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:58.677431    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbndh\" (UniqueName: \"kubernetes.io/projected/a67a0d69-b558-40d4-80de-0928d6ce5fb9-kube-api-access-pbndh\") pod \"test-curl-7609-3410cd\" (UID: \"a67a0d69-b558-40d4-80de-0928d6ce5fb9\") " pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:02:58.779740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:58.779697    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbndh\" (UniqueName: \"kubernetes.io/projected/a67a0d69-b558-40d4-80de-0928d6ce5fb9-kube-api-access-pbndh\") pod \"test-curl-7609-3410cd\" (UID: \"a67a0d69-b558-40d4-80de-0928d6ce5fb9\") " pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:02:58.788691 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:58.788664    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbndh\" (UniqueName: \"kubernetes.io/projected/a67a0d69-b558-40d4-80de-0928d6ce5fb9-kube-api-access-pbndh\") pod \"test-curl-7609-3410cd\" (UID: \"a67a0d69-b558-40d4-80de-0928d6ce5fb9\") " pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:02:58.892112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:58.892079    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:02:59.173586 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:02:59.173556    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda67a0d69_b558_40d4_80de_0928d6ce5fb9.slice/crio-c1269fbec5cde0d8b64c20ef7261bbaa37d8bd12f660d50bfb8a7516b57fde89 WatchSource:0}: Error finding container c1269fbec5cde0d8b64c20ef7261bbaa37d8bd12f660d50bfb8a7516b57fde89: Status 404 returned error can't find the container with id c1269fbec5cde0d8b64c20ef7261bbaa37d8bd12f660d50bfb8a7516b57fde89
Nov 03 11:02:59.176000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:02:59.175976    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-3410cd"]
Nov 03 11:03:00.086390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:00.086345    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-3410cd" event={"ID":"a67a0d69-b558-40d4-80de-0928d6ce5fb9","Type":"ContainerStarted","Data":"9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce"}
Nov 03 11:03:00.087941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:00.087895    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-3410cd" event={"ID":"a67a0d69-b558-40d4-80de-0928d6ce5fb9","Type":"ContainerStarted","Data":"c1269fbec5cde0d8b64c20ef7261bbaa37d8bd12f660d50bfb8a7516b57fde89"}
Nov 03 11:03:00.126957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:00.126853    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-3410cd" podStartSLOduration=2.126833636 podStartE2EDuration="2.126833636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:00.119475648 +0000 UTC m=+2024.024330090" watchObservedRunningTime="2026-11-03 11:03:00.126833636 +0000 UTC m=+2024.031688082"
Nov 03 11:03:00.792532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:00.792493    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-3410cd"]
Nov 03 11:03:00.799563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:00.799526    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-3410cd"]
Nov 03 11:03:02.133354 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:02.133309    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-3410cd" podUID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" containerName="test-curl-7609-3410cd" containerID="cri-o://9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce" gracePeriod=2
Nov 03 11:03:02.138088 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:03:02.138052    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7609-3410cd\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:03:04.369882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:04.369857    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:03:04.491706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:04.491610    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67a0d69-b558-40d4-80de-0928d6ce5fb9-kube-api-access-pbndh\" (UniqueName: \"kubernetes.io/projected/a67a0d69-b558-40d4-80de-0928d6ce5fb9-kube-api-access-pbndh\") pod \"a67a0d69-b558-40d4-80de-0928d6ce5fb9\" (UID: \"a67a0d69-b558-40d4-80de-0928d6ce5fb9\") "
Nov 03 11:03:04.494067 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:04.494031    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67a0d69-b558-40d4-80de-0928d6ce5fb9-kube-api-access-pbndh" pod "a67a0d69-b558-40d4-80de-0928d6ce5fb9" (UID: "a67a0d69-b558-40d4-80de-0928d6ce5fb9"). InnerVolumeSpecName "kube-api-access-pbndh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:03:04.594003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:04.593957    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbndh\" (UniqueName: \"kubernetes.io/projected/a67a0d69-b558-40d4-80de-0928d6ce5fb9-kube-api-access-pbndh\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:05.192601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:05.192555    5393 generic.go:356] "Generic (PLEG): container finished" podID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" containerID="9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce" exitCode=137
Nov 03 11:03:05.192601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:05.192601    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:03:05.192854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:05.192660    5393 scope.go:122] "RemoveContainer" containerID="9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce"
Nov 03 11:03:05.209227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:05.209108    5393 scope.go:122] "RemoveContainer" containerID="9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce"
Nov 03 11:03:05.210014 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:03:05.209984    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce\": container with ID starting with 9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce not found: ID does not exist" containerID="9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce"
Nov 03 11:03:05.210122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:05.210022    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce"} err="failed to get container status \"9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce\": rpc error: code = NotFound desc = could not find container \"9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce\": container with ID starting with 9a4f87bc9c946d0eed622e70b446da981237a92c5b60b8a79fd33d3d7cbf68ce not found: ID does not exist"
Nov 03 11:03:05.212134 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:03:05.212107    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7609-3410cd\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:03:06.229767 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:03:06.229722    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7609-3410cd\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" pod="opendatahub/test-curl-7609-3410cd"
Nov 03 11:03:06.232663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:06.232633    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" path="/var/lib/kubelet/pods/a67a0d69-b558-40d4-80de-0928d6ce5fb9/volumes"
Nov 03 11:03:13.190613 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:03:13.190569    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:03:18.509037 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.509005    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"]
Nov 03 11:03:18.509857 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.509829    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" containerName="test-curl-7609-3410cd"
Nov 03 11:03:18.509857 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.509854    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" containerName="test-curl-7609-3410cd"
Nov 03 11:03:18.510057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.510000    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67a0d69-b558-40d4-80de-0928d6ce5fb9" containerName="test-curl-7609-3410cd"
Nov 03 11:03:18.514055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.514036    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.524863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.524828    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d9f2ce-openshift-default-dockercfg-zldhk\""
Nov 03 11:03:18.538579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.538550    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"]
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.634858    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.635020    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.635084    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcjxm\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-kube-api-access-jcjxm\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.635176    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.635213    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.635415    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.635469    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.635508    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.637945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.635533    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.742378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.742344    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.745945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.742571    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.746235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.746214    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.746394 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.746380    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.746527 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.746511    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcjxm\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-kube-api-access-jcjxm\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.746673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.746661    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.761001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.746752    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.761516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.761481    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.761715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.761697    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.762723 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.762696    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.762967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.743188    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.763454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.763429    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.763818 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.763800    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.775951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.772704    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.800032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.799989    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.801615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.801589    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.802782 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.802721    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcjxm\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-kube-api-access-jcjxm\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:18.899150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:18.899088    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:19.130543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:19.130512    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:19.433146 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:19.433115    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"]
Nov 03 11:03:19.433285 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:03:19.433255    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae47eae1_5e93_49ab_b252_7e5c30dd1d23.slice/crio-1fd6b46010a6dc7a0eeef87bcaf005a0d5b1aabd4aea13ab1772bfbe4b78f0c9 WatchSource:0}: Error finding container 1fd6b46010a6dc7a0eeef87bcaf005a0d5b1aabd4aea13ab1772bfbe4b78f0c9: Status 404 returned error can't find the container with id 1fd6b46010a6dc7a0eeef87bcaf005a0d5b1aabd4aea13ab1772bfbe4b78f0c9
Nov 03 11:03:19.436325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:19.436275    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:19.436436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:19.436351    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:19.436436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:19.436389    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:03:20.341184 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:20.341151    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" event={"ID":"ae47eae1-5e93-49ab-b252-7e5c30dd1d23","Type":"ContainerStarted","Data":"8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306"}
Nov 03 11:03:20.341695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:20.341201    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" event={"ID":"ae47eae1-5e93-49ab-b252-7e5c30dd1d23","Type":"ContainerStarted","Data":"1fd6b46010a6dc7a0eeef87bcaf005a0d5b1aabd4aea13ab1772bfbe4b78f0c9"}
Nov 03 11:03:20.370988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:20.370934    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podStartSLOduration=2.370907107 podStartE2EDuration="2.370907107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:20.366045163 +0000 UTC m=+2044.270899604" watchObservedRunningTime="2026-11-03 11:03:20.370907107 +0000 UTC m=+2044.275761624"
Nov 03 11:03:21.131387 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:21.131352    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:21.137336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:21.137310    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:21.349393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:21.349360    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:21.350498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:21.350476    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:03:23.016511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:23.016472    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"]
Nov 03 11:03:28.255942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:28.254148    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"]
Nov 03 11:03:34.106950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.106902    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"]
Nov 03 11:03:34.112122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.112084    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:34.119551 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.119526    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d9f2ce-dockercfg-6ntgt\""
Nov 03 11:03:34.119851 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.119813    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d9f2ce\""
Nov 03 11:03:34.124638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.124610    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"]
Nov 03 11:03:34.310943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.285967    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vc89\" (UniqueName: \"kubernetes.io/projected/d3694a1b-be9a-4684-b795-ae49922bf43c-kube-api-access-6vc89\") pod \"payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s\" (UID: \"d3694a1b-be9a-4684-b795-ae49922bf43c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:34.311705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.311671    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3694a1b-be9a-4684-b795-ae49922bf43c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s\" (UID: \"d3694a1b-be9a-4684-b795-ae49922bf43c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:34.413530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.413444    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3694a1b-be9a-4684-b795-ae49922bf43c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s\" (UID: \"d3694a1b-be9a-4684-b795-ae49922bf43c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:34.413840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.413815    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vc89\" (UniqueName: \"kubernetes.io/projected/d3694a1b-be9a-4684-b795-ae49922bf43c-kube-api-access-6vc89\") pod \"payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s\" (UID: \"d3694a1b-be9a-4684-b795-ae49922bf43c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:34.414389 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.414367    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3694a1b-be9a-4684-b795-ae49922bf43c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s\" (UID: \"d3694a1b-be9a-4684-b795-ae49922bf43c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:34.451435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.441436    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vc89\" (UniqueName: \"kubernetes.io/projected/d3694a1b-be9a-4684-b795-ae49922bf43c-kube-api-access-6vc89\") pod \"payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s\" (UID: \"d3694a1b-be9a-4684-b795-ae49922bf43c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:34.739732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:34.739648    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:35.027224 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:03:35.027193    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3694a1b_be9a_4684_b795_ae49922bf43c.slice/crio-9c321b5910f78f4b71a54f4709aaca201574c491a6dbebea8e4fa06ec38efb35 WatchSource:0}: Error finding container 9c321b5910f78f4b71a54f4709aaca201574c491a6dbebea8e4fa06ec38efb35: Status 404 returned error can't find the container with id 9c321b5910f78f4b71a54f4709aaca201574c491a6dbebea8e4fa06ec38efb35
Nov 03 11:03:35.027447 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:35.027427    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"]
Nov 03 11:03:35.505623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:35.505587    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s" event={"ID":"d3694a1b-be9a-4684-b795-ae49922bf43c","Type":"ContainerStarted","Data":"67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d"}
Nov 03 11:03:35.506098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:35.505643    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s" event={"ID":"d3694a1b-be9a-4684-b795-ae49922bf43c","Type":"ContainerStarted","Data":"9c321b5910f78f4b71a54f4709aaca201574c491a6dbebea8e4fa06ec38efb35"}
Nov 03 11:03:35.506098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:35.505681    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:35.526027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:35.525974    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s" podStartSLOduration=1.5259612200000001 podStartE2EDuration="1.52596122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:35.522849497 +0000 UTC m=+2059.427703940" watchObservedRunningTime="2026-11-03 11:03:35.52596122 +0000 UTC m=+2059.430815661"
Nov 03 11:03:38.049437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.049385    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w" podUID="1b69c7ad-8e31-42c8-922d-b9dab996e617" containerName="main" containerID="cri-o://828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142" gracePeriod=45
Nov 03 11:03:38.342678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.342652    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 11:03:38.490861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.490831    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b69c7ad-8e31-42c8-922d-b9dab996e617-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b69c7ad-8e31-42c8-922d-b9dab996e617-tls-certs\") pod \"1b69c7ad-8e31-42c8-922d-b9dab996e617\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") "
Nov 03 11:03:38.491063 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.490930    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-model-cache\") pod \"1b69c7ad-8e31-42c8-922d-b9dab996e617\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") "
Nov 03 11:03:38.491063 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.490995    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b69c7ad-8e31-42c8-922d-b9dab996e617-kube-api-access-ctxwm\" (UniqueName: \"kubernetes.io/projected/1b69c7ad-8e31-42c8-922d-b9dab996e617-kube-api-access-ctxwm\") pod \"1b69c7ad-8e31-42c8-922d-b9dab996e617\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") "
Nov 03 11:03:38.491063 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.491053    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-dshm\") pod \"1b69c7ad-8e31-42c8-922d-b9dab996e617\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") "
Nov 03 11:03:38.491264 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.491129    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-home\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-home\") pod \"1b69c7ad-8e31-42c8-922d-b9dab996e617\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") "
Nov 03 11:03:38.491264 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.491174    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-tmp-dir\") pod \"1b69c7ad-8e31-42c8-922d-b9dab996e617\" (UID: \"1b69c7ad-8e31-42c8-922d-b9dab996e617\") "
Nov 03 11:03:38.491264 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.491198    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-model-cache" pod "1b69c7ad-8e31-42c8-922d-b9dab996e617" (UID: "1b69c7ad-8e31-42c8-922d-b9dab996e617"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:03:38.491444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.491418    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-home" pod "1b69c7ad-8e31-42c8-922d-b9dab996e617" (UID: "1b69c7ad-8e31-42c8-922d-b9dab996e617"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:03:38.491531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.491510    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-tmp-dir" pod "1b69c7ad-8e31-42c8-922d-b9dab996e617" (UID: "1b69c7ad-8e31-42c8-922d-b9dab996e617"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:03:38.492028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.492009    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:38.492139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.492032    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:38.492139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.492045    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:38.493523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.493500    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b69c7ad-8e31-42c8-922d-b9dab996e617-tls-certs" pod "1b69c7ad-8e31-42c8-922d-b9dab996e617" (UID: "1b69c7ad-8e31-42c8-922d-b9dab996e617"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:03:38.493634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.493532    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b69c7ad-8e31-42c8-922d-b9dab996e617-kube-api-access-ctxwm" pod "1b69c7ad-8e31-42c8-922d-b9dab996e617" (UID: "1b69c7ad-8e31-42c8-922d-b9dab996e617"). InnerVolumeSpecName "kube-api-access-ctxwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:03:38.493634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.493533    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-dshm" pod "1b69c7ad-8e31-42c8-922d-b9dab996e617" (UID: "1b69c7ad-8e31-42c8-922d-b9dab996e617"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:03:38.533776 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.533734    5393 generic.go:356] "Generic (PLEG): container finished" podID="1b69c7ad-8e31-42c8-922d-b9dab996e617" containerID="828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142" exitCode=0
Nov 03 11:03:38.533973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.533789    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"
Nov 03 11:03:38.533973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.533814    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w" event={"ID":"1b69c7ad-8e31-42c8-922d-b9dab996e617","Type":"ContainerDied","Data":"828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142"}
Nov 03 11:03:38.533973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.533856    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w" event={"ID":"1b69c7ad-8e31-42c8-922d-b9dab996e617","Type":"ContainerDied","Data":"a5b129db78112600cd637f8cb04a97e70f326ae734d97246dd11d7630cbd1f4f"}
Nov 03 11:03:38.533973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.533873    5393 scope.go:122] "RemoveContainer" containerID="828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142"
Nov 03 11:03:38.549385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.549365    5393 scope.go:122] "RemoveContainer" containerID="828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142"
Nov 03 11:03:38.549688 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:03:38.549670    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142\": container with ID starting with 828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142 not found: ID does not exist" containerID="828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142"
Nov 03 11:03:38.549740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.549699    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142"} err="failed to get container status \"828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142\": rpc error: code = NotFound desc = could not find container \"828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142\": container with ID starting with 828c4d27db911657e033a8cbd909733014b1ac4002f78c4d3986e08e63079142 not found: ID does not exist"
Nov 03 11:03:38.565629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.565564    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"]
Nov 03 11:03:38.573183 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.573156    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce5555/auth-test-model-ce5555-kserve-5744d85cb9-trs2w"]
Nov 03 11:03:38.594271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.594241    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxwm\" (UniqueName: \"kubernetes.io/projected/1b69c7ad-8e31-42c8-922d-b9dab996e617-kube-api-access-ctxwm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:38.594271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.594266    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b69c7ad-8e31-42c8-922d-b9dab996e617-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:38.594426 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:38.594278    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b69c7ad-8e31-42c8-922d-b9dab996e617-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:40.244009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:40.243521    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b69c7ad-8e31-42c8-922d-b9dab996e617" path="/var/lib/kubelet/pods/1b69c7ad-8e31-42c8-922d-b9dab996e617/volumes"
Nov 03 11:03:41.853112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.851982    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"]
Nov 03 11:03:41.855022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.853901    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1b69c7ad-8e31-42c8-922d-b9dab996e617" containerName="main"
Nov 03 11:03:41.855022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.853945    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b69c7ad-8e31-42c8-922d-b9dab996e617" containerName="main"
Nov 03 11:03:41.855022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.854217    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b69c7ad-8e31-42c8-922d-b9dab996e617" containerName="main"
Nov 03 11:03:41.868842 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.868814    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:41.879598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.875853    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"]
Nov 03 11:03:41.879598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.878106    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-e40ddfc7cf41\""
Nov 03 11:03:41.879598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.878166    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-586b7631a686\""
Nov 03 11:03:41.879598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.878239    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"openshift-service-ca.crt\""
Nov 03 11:03:41.879598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.878297    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"kube-root-ca.crt\""
Nov 03 11:03:41.879598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.878329    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fb5766\""
Nov 03 11:03:41.879598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.878390    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-e2e--b8bff1be-dockercfg-45kcv\""
Nov 03 11:03:41.887035 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:41.885491    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"routing-overlay\""
Nov 03 11:03:42.035529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.033278    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqb8p\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-kube-api-access-zqb8p\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.035529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.033491    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.035529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.033538    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.035529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.033687    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.135448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.135345    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.135640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.135485    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqb8p\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-kube-api-access-zqb8p\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.135709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.135668    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.136422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.136392    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.136568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.136540    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.136629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.136601    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.138904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.138876    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.148615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.148587    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqb8p\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-kube-api-access-zqb8p\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.218582 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.218542    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"]
Nov 03 11:03:42.223432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.223408    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.239070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.239043    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"]
Nov 03 11:03:42.322882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.322850    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56687c\"/\"agc-cred-eeeff88d5e57\""
Nov 03 11:03:42.339659 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.339631    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8gx\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-kube-api-access-cj8gx\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.339776 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.339680    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.340200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.340173    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.340293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.340274    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.394280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.394188    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:42.442300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.442263    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.442495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.442328    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.442576 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.442506    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8gx\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-kube-api-access-cj8gx\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.442576 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.442539    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.443195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.443169    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.443363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.443334    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.445380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.445355    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.451662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.451639    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8gx\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-kube-api-access-cj8gx\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.544556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.544522    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:42.704999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.704971    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"]
Nov 03 11:03:42.707321 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:03:42.707295    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bca7ce_baf9_4911_ab93_12009db06a6d.slice/crio-38bc3a766cdd4234f7da0f923b4184a714d60e161920ad268e21f0246a0d7832 WatchSource:0}: Error finding container 38bc3a766cdd4234f7da0f923b4184a714d60e161920ad268e21f0246a0d7832: Status 404 returned error can't find the container with id 38bc3a766cdd4234f7da0f923b4184a714d60e161920ad268e21f0246a0d7832
Nov 03 11:03:42.896142 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:03:42.896077    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beccb1c_3d58_41fa_8689_f416f57e968e.slice/crio-48fcfc498d9d126c1f4d289438f77afea092d5b65012f0f37ea409cbcc7505ce WatchSource:0}: Error finding container 48fcfc498d9d126c1f4d289438f77afea092d5b65012f0f37ea409cbcc7505ce: Status 404 returned error can't find the container with id 48fcfc498d9d126c1f4d289438f77afea092d5b65012f0f37ea409cbcc7505ce
Nov 03 11:03:42.913956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:42.913896    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"]
Nov 03 11:03:43.191358 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:03:43.191314    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:03:43.282040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.281911    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j" podUID="877bd7e3-b64c-45e0-a339-2a0574e99bef" containerName="main" containerID="cri-o://b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181" gracePeriod=45
Nov 03 11:03:43.576897 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.576867    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 11:03:43.581398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.581367    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7" event={"ID":"3beccb1c-3d58-41fa-8689-f416f57e968e","Type":"ContainerStarted","Data":"f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef"}
Nov 03 11:03:43.581574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.581432    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:43.581574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.581457    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7" event={"ID":"3beccb1c-3d58-41fa-8689-f416f57e968e","Type":"ContainerStarted","Data":"48fcfc498d9d126c1f4d289438f77afea092d5b65012f0f37ea409cbcc7505ce"}
Nov 03 11:03:43.583226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.583195    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7" event={"ID":"65bca7ce-baf9-4911-ab93-12009db06a6d","Type":"ContainerStarted","Data":"1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0"}
Nov 03 11:03:43.583366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.583247    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:43.583366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.583263    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7" event={"ID":"65bca7ce-baf9-4911-ab93-12009db06a6d","Type":"ContainerStarted","Data":"38bc3a766cdd4234f7da0f923b4184a714d60e161920ad268e21f0246a0d7832"}
Nov 03 11:03:43.585025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.584999    5393 generic.go:356] "Generic (PLEG): container finished" podID="877bd7e3-b64c-45e0-a339-2a0574e99bef" containerID="b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181" exitCode=0
Nov 03 11:03:43.585136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.585032    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"
Nov 03 11:03:43.585136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.585074    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j" event={"ID":"877bd7e3-b64c-45e0-a339-2a0574e99bef","Type":"ContainerDied","Data":"b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181"}
Nov 03 11:03:43.585136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.585110    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j" event={"ID":"877bd7e3-b64c-45e0-a339-2a0574e99bef","Type":"ContainerDied","Data":"5dc1a8e5fcf5e418263ec10e3e0cc4ec7a587603a88d50552ef1f762705c3e49"}
Nov 03 11:03:43.585136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.585132    5393 scope.go:122] "RemoveContainer" containerID="b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181"
Nov 03 11:03:43.597802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.597777    5393 scope.go:122] "RemoveContainer" containerID="b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181"
Nov 03 11:03:43.598205 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:03:43.598177    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181\": container with ID starting with b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181 not found: ID does not exist" containerID="b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181"
Nov 03 11:03:43.598342 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.598210    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181"} err="failed to get container status \"b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181\": rpc error: code = NotFound desc = could not find container \"b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181\": container with ID starting with b8cf39e841ae9472288b921d94215256fc26fc3776c16a90c583b92499d4e181 not found: ID does not exist"
Nov 03 11:03:43.647498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.647440    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7" podStartSLOduration=1.647421282 podStartE2EDuration="1.647421282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:43.645818196 +0000 UTC m=+2067.550672641" watchObservedRunningTime="2026-11-03 11:03:43.647421282 +0000 UTC m=+2067.552275723"
Nov 03 11:03:43.672814 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.672786    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-home\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-home\") pod \"877bd7e3-b64c-45e0-a339-2a0574e99bef\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") "
Nov 03 11:03:43.673062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.672957    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877bd7e3-b64c-45e0-a339-2a0574e99bef-kube-api-access-zqszh\" (UniqueName: \"kubernetes.io/projected/877bd7e3-b64c-45e0-a339-2a0574e99bef-kube-api-access-zqszh\") pod \"877bd7e3-b64c-45e0-a339-2a0574e99bef\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") "
Nov 03 11:03:43.673139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.673101    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-dshm\") pod \"877bd7e3-b64c-45e0-a339-2a0574e99bef\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") "
Nov 03 11:03:43.673139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.673111    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-home" pod "877bd7e3-b64c-45e0-a339-2a0574e99bef" (UID: "877bd7e3-b64c-45e0-a339-2a0574e99bef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:03:43.673249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.673180    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-tmp-dir\") pod \"877bd7e3-b64c-45e0-a339-2a0574e99bef\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") "
Nov 03 11:03:43.673303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.673251    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-model-cache\") pod \"877bd7e3-b64c-45e0-a339-2a0574e99bef\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") "
Nov 03 11:03:43.673392 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.673376    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/877bd7e3-b64c-45e0-a339-2a0574e99bef-tls-certs\" (UniqueName: \"kubernetes.io/secret/877bd7e3-b64c-45e0-a339-2a0574e99bef-tls-certs\") pod \"877bd7e3-b64c-45e0-a339-2a0574e99bef\" (UID: \"877bd7e3-b64c-45e0-a339-2a0574e99bef\") "
Nov 03 11:03:43.673454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.673435    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-tmp-dir" pod "877bd7e3-b64c-45e0-a339-2a0574e99bef" (UID: "877bd7e3-b64c-45e0-a339-2a0574e99bef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:03:43.673731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.673600    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-model-cache" pod "877bd7e3-b64c-45e0-a339-2a0574e99bef" (UID: "877bd7e3-b64c-45e0-a339-2a0574e99bef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:03:43.674287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.674268    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:43.674402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.674290    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:43.674402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.674303    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:43.675505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.675485    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-dshm" pod "877bd7e3-b64c-45e0-a339-2a0574e99bef" (UID: "877bd7e3-b64c-45e0-a339-2a0574e99bef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:03:43.675614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.675549    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877bd7e3-b64c-45e0-a339-2a0574e99bef-kube-api-access-zqszh" pod "877bd7e3-b64c-45e0-a339-2a0574e99bef" (UID: "877bd7e3-b64c-45e0-a339-2a0574e99bef"). InnerVolumeSpecName "kube-api-access-zqszh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:03:43.675766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.675752    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877bd7e3-b64c-45e0-a339-2a0574e99bef-tls-certs" pod "877bd7e3-b64c-45e0-a339-2a0574e99bef" (UID: "877bd7e3-b64c-45e0-a339-2a0574e99bef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:03:43.678958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.678888    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7" podStartSLOduration=2.678875899 podStartE2EDuration="2.678875899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:43.676262404 +0000 UTC m=+2067.581116846" watchObservedRunningTime="2026-11-03 11:03:43.678875899 +0000 UTC m=+2067.583730339"
Nov 03 11:03:43.776409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.776369    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqszh\" (UniqueName: \"kubernetes.io/projected/877bd7e3-b64c-45e0-a339-2a0574e99bef-kube-api-access-zqszh\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:43.776409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.776404    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877bd7e3-b64c-45e0-a339-2a0574e99bef-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:43.776409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.776416    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877bd7e3-b64c-45e0-a339-2a0574e99bef-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:03:43.915736 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.915619    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"]
Nov 03 11:03:43.919936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:43.919896    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/auth-test-model-836589-kserve-6bfb4cbcc7-wlx9j"]
Nov 03 11:03:44.233208 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.233122    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877bd7e3-b64c-45e0-a339-2a0574e99bef" path="/var/lib/kubelet/pods/877bd7e3-b64c-45e0-a339-2a0574e99bef/volumes"
Nov 03 11:03:44.933851 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.933815    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"]
Nov 03 11:03:44.934855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.934839    5393 cpu_manager.go:414] "containerMap: removing container" podUID="877bd7e3-b64c-45e0-a339-2a0574e99bef" containerName="main"
Nov 03 11:03:44.934898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.934858    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877bd7e3-b64c-45e0-a339-2a0574e99bef" containerName="main"
Nov 03 11:03:44.935050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.935038    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="877bd7e3-b64c-45e0-a339-2a0574e99bef" containerName="main"
Nov 03 11:03:44.940419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.940392    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:44.953185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.953159    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c8cb1f\""
Nov 03 11:03:44.953338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.953296    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c8cb1f-dockercfg-nwxf7\""
Nov 03 11:03:44.958269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.958241    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"]
Nov 03 11:03:44.964378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.964354    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"]
Nov 03 11:03:44.970842 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.970820    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:44.984525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:44.982479    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"]
Nov 03 11:03:45.263992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.263458    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntjpj\" (UniqueName: \"kubernetes.io/projected/9485a2e7-7f1d-4d4f-b461-261daa51e5de-kube-api-access-ntjpj\") pod \"payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6\" (UID: \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:45.263992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.263530    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9485a2e7-7f1d-4d4f-b461-261daa51e5de-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6\" (UID: \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:45.263992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.263691    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkf4f\" (UniqueName: \"kubernetes.io/projected/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-kube-api-access-pkf4f\") pod \"payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2\" (UID: \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\") " pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:45.263992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.263758    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-plugins-config-volume\") pod \"payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2\" (UID: \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\") " pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:45.365439 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.365396    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkf4f\" (UniqueName: \"kubernetes.io/projected/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-kube-api-access-pkf4f\") pod \"payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2\" (UID: \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\") " pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:45.365632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.365454    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-plugins-config-volume\") pod \"payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2\" (UID: \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\") " pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:45.365632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.365601    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjpj\" (UniqueName: \"kubernetes.io/projected/9485a2e7-7f1d-4d4f-b461-261daa51e5de-kube-api-access-ntjpj\") pod \"payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6\" (UID: \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:45.365632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.365627    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9485a2e7-7f1d-4d4f-b461-261daa51e5de-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6\" (UID: \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:45.366246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.366223    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-plugins-config-volume\") pod \"payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2\" (UID: \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\") " pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:45.366351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.366334    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9485a2e7-7f1d-4d4f-b461-261daa51e5de-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6\" (UID: \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:45.377507 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.377477    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjpj\" (UniqueName: \"kubernetes.io/projected/9485a2e7-7f1d-4d4f-b461-261daa51e5de-kube-api-access-ntjpj\") pod \"payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6\" (UID: \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:45.377646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.377630    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkf4f\" (UniqueName: \"kubernetes.io/projected/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-kube-api-access-pkf4f\") pod \"payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2\" (UID: \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\") " pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:45.556240 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.556153    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:45.619939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.610256    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:45.935723 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.935694    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"]
Nov 03 11:03:45.936193 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:03:45.936169    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0b28aa_6951_4fbb_aefa_08cf5c86ae43.slice/crio-0fbc869ece8113de8ad37a9802e873c99d811c716c003f50694f9d14e4587e04 WatchSource:0}: Error finding container 0fbc869ece8113de8ad37a9802e873c99d811c716c003f50694f9d14e4587e04: Status 404 returned error can't find the container with id 0fbc869ece8113de8ad37a9802e873c99d811c716c003f50694f9d14e4587e04
Nov 03 11:03:45.961254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:45.961202    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"]
Nov 03 11:03:45.962235 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:03:45.962208    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9485a2e7_7f1d_4d4f_b461_261daa51e5de.slice/crio-67441fde6b8baa64f5a3f584b593830d5236ac51a96fd4a7c320997d5d6884fe WatchSource:0}: Error finding container 67441fde6b8baa64f5a3f584b593830d5236ac51a96fd4a7c320997d5d6884fe: Status 404 returned error can't find the container with id 67441fde6b8baa64f5a3f584b593830d5236ac51a96fd4a7c320997d5d6884fe
Nov 03 11:03:46.519435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.519408    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:03:46.664721 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.664634    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2" event={"ID":"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43","Type":"ContainerStarted","Data":"81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95"}
Nov 03 11:03:46.664936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.664784    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2" event={"ID":"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43","Type":"ContainerStarted","Data":"0fbc869ece8113de8ad37a9802e873c99d811c716c003f50694f9d14e4587e04"}
Nov 03 11:03:46.665049 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.665018    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:46.669268 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.669239    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6" event={"ID":"9485a2e7-7f1d-4d4f-b461-261daa51e5de","Type":"ContainerStarted","Data":"fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002"}
Nov 03 11:03:46.669477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.669293    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6" event={"ID":"9485a2e7-7f1d-4d4f-b461-261daa51e5de","Type":"ContainerStarted","Data":"67441fde6b8baa64f5a3f584b593830d5236ac51a96fd4a7c320997d5d6884fe"}
Nov 03 11:03:46.669477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.669400    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:46.714314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.714258    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2" podStartSLOduration=2.714242595 podStartE2EDuration="2.714242595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:46.710032731 +0000 UTC m=+2070.614887172" watchObservedRunningTime="2026-11-03 11:03:46.714242595 +0000 UTC m=+2070.619097036"
Nov 03 11:03:46.743254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:46.743188    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6" podStartSLOduration=2.743167791 podStartE2EDuration="2.743167791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:03:46.740806854 +0000 UTC m=+2070.645661293" watchObservedRunningTime="2026-11-03 11:03:46.743167791 +0000 UTC m=+2070.648022234"
Nov 03 11:03:54.597253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:54.597223    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:03:54.597722 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:54.597299    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:03:54.653835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:54.653357    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"]
Nov 03 11:03:57.696615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:57.695325    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:03:57.698650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:57.698625    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:03:59.654364 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:59.654311    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d" podUID="eed5d139-d3c3-4739-ad7a-9be8852efe9d" containerName="payload-processing" containerID="cri-o://7f3acded20c8ead3ee39737ab491b04a0d117c0b1f631cf940aa739037ae819d" gracePeriod=25
Nov 03 11:03:59.835216 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:59.835175    5393 generic.go:356] "Generic (PLEG): container finished" podID="eed5d139-d3c3-4739-ad7a-9be8852efe9d" containerID="7f3acded20c8ead3ee39737ab491b04a0d117c0b1f631cf940aa739037ae819d" exitCode=0
Nov 03 11:03:59.835415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:59.835250    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d" event={"ID":"eed5d139-d3c3-4739-ad7a-9be8852efe9d","Type":"ContainerDied","Data":"7f3acded20c8ead3ee39737ab491b04a0d117c0b1f631cf940aa739037ae819d"}
Nov 03 11:03:59.994614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:03:59.994585    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:04:00.136775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.136738    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-plugins-config-volume\") pod \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") "
Nov 03 11:04:00.137050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.136903    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-routing-overlay-volume\") pod \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") "
Nov 03 11:04:00.137050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.136994    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-kube-api-access-z78tn\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-kube-api-access-z78tn\") pod \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") "
Nov 03 11:04:00.137210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.137110    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-provider-credentials\") pod \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\" (UID: \"eed5d139-d3c3-4739-ad7a-9be8852efe9d\") "
Nov 03 11:04:00.137326 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.137300    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-plugins-config-volume" pod "eed5d139-d3c3-4739-ad7a-9be8852efe9d" (UID: "eed5d139-d3c3-4739-ad7a-9be8852efe9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:00.137395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.137356    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-routing-overlay-volume" pod "eed5d139-d3c3-4739-ad7a-9be8852efe9d" (UID: "eed5d139-d3c3-4739-ad7a-9be8852efe9d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:00.137691 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.137671    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:00.137767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.137695    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eed5d139-d3c3-4739-ad7a-9be8852efe9d-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:00.139432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.139399    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-kube-api-access-z78tn" pod "eed5d139-d3c3-4739-ad7a-9be8852efe9d" (UID: "eed5d139-d3c3-4739-ad7a-9be8852efe9d"). InnerVolumeSpecName "kube-api-access-z78tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:00.139529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.139451    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-provider-credentials" pod "eed5d139-d3c3-4739-ad7a-9be8852efe9d" (UID: "eed5d139-d3c3-4739-ad7a-9be8852efe9d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:00.239312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.239227    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:00.239312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.239262    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z78tn\" (UniqueName: \"kubernetes.io/projected/eed5d139-d3c3-4739-ad7a-9be8852efe9d-kube-api-access-z78tn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:00.844820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.844785    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"
Nov 03 11:04:00.844820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.844809    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d" event={"ID":"eed5d139-d3c3-4739-ad7a-9be8852efe9d","Type":"ContainerDied","Data":"3907cb9f14764f014f131bcd860319c5f233ac5fc844424621c01d6a9b7c1d52"}
Nov 03 11:04:00.845407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.844867    5393 scope.go:122] "RemoveContainer" containerID="7f3acded20c8ead3ee39737ab491b04a0d117c0b1f631cf940aa739037ae819d"
Nov 03 11:04:00.865829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.865802    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"]
Nov 03 11:04:00.869487 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:00.869457    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbt5n6d"]
Nov 03 11:04:01.676889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.676857    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"]
Nov 03 11:04:01.677577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.677562    5393 cpu_manager.go:414] "containerMap: removing container" podUID="eed5d139-d3c3-4739-ad7a-9be8852efe9d" containerName="payload-processing"
Nov 03 11:04:01.677629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.677579    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed5d139-d3c3-4739-ad7a-9be8852efe9d" containerName="payload-processing"
Nov 03 11:04:01.677727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.677718    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed5d139-d3c3-4739-ad7a-9be8852efe9d" containerName="payload-processing"
Nov 03 11:04:01.683295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.683269    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.691676 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.690142    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c8cb1f-serving-cert\""
Nov 03 11:04:01.698967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.698938    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"]
Nov 03 11:04:01.760490 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.760460    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-maas-api-tls\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.760670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.760509    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e2ee79-5674-4b90-b477-eb8635a1e51c-openshift-service-ca\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.760670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.760652    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgkx\" (UniqueName: \"kubernetes.io/projected/a6e2ee79-5674-4b90-b477-eb8635a1e51c-kube-api-access-fwgkx\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.760746 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.760695    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-metrics-certs\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.862124 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.862088    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-maas-api-tls\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.862624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.862134    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e2ee79-5674-4b90-b477-eb8635a1e51c-openshift-service-ca\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.862624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.862315    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgkx\" (UniqueName: \"kubernetes.io/projected/a6e2ee79-5674-4b90-b477-eb8635a1e51c-kube-api-access-fwgkx\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.862624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.862357    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-metrics-certs\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.862860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.862842    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e2ee79-5674-4b90-b477-eb8635a1e51c-openshift-service-ca\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.865037 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.865012    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-maas-api-tls\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.865161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.865075    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-metrics-certs\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:01.871296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:01.871268    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgkx\" (UniqueName: \"kubernetes.io/projected/a6e2ee79-5674-4b90-b477-eb8635a1e51c-kube-api-access-fwgkx\") pod \"maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:02.013342 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:02.013253    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:02.328992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:02.326590    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed5d139-d3c3-4739-ad7a-9be8852efe9d" path="/var/lib/kubelet/pods/eed5d139-d3c3-4739-ad7a-9be8852efe9d/volumes"
Nov 03 11:04:02.374298 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:02.370815    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"]
Nov 03 11:04:02.862448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:02.862417    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj" event={"ID":"a6e2ee79-5674-4b90-b477-eb8635a1e51c","Type":"ContainerStarted","Data":"7336a134671051e822c8b09212e0ecb568fbea796cab17ea6bda16c88e733c50"}
Nov 03 11:04:03.873330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:03.873275    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj" event={"ID":"a6e2ee79-5674-4b90-b477-eb8635a1e51c","Type":"ContainerStarted","Data":"7fb43337315cb37e98250f84ab821a658d8ef197ebbf418874826f808f0bf076"}
Nov 03 11:04:03.873786 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:03.873352    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:03.892528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:03.892478    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj" podStartSLOduration=2.463792056 podStartE2EDuration="2.892461927s" totalImagesPullingTime="428.669871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:03.891486067 +0000 UTC m=+2087.796340509" watchObservedRunningTime="2026-11-03 11:04:03.892461927 +0000 UTC m=+2087.797316368"
Nov 03 11:04:09.900132 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:09.900096    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:04:13.212797 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:04:13.212758    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:04:17.473330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:17.473299    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:04:17.473784 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:17.473356    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:04:18.612395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.612361    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"]
Nov 03 11:04:18.618063 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.618034    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.643209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.643181    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"]
Nov 03 11:04:18.780619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.780575    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.780800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.780640    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.780800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.780762    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.780957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.780884    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.781026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.780962    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.781068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.781042    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.781108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.781087    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bg4\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-kube-api-access-d8bg4\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.781223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.781206    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.781295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.781283    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883188    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883240    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bg4\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-kube-api-access-d8bg4\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883307    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883360    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883432    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883490    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883538    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883595    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883624    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883662    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883708    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.883982 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.883854    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.884045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.884007    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.884287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.884262    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.886074 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.886055    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.886291 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.886274    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.892479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.892450    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bg4\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-kube-api-access-d8bg4\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.892658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.892640    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:18.987973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:18.987229    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:19.126025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.120941    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" event={"ID":"ae47eae1-5e93-49ab-b252-7e5c30dd1d23","Type":"ContainerDied","Data":"8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306"}
Nov 03 11:04:19.126699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.126667    5393 scope.go:122] "RemoveContainer" containerID="8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306"
Nov 03 11:04:19.129896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.120876    5393 generic.go:356] "Generic (PLEG): container finished" podID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerID="8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306" exitCode=0
Nov 03 11:04:19.133882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.133299    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:19.133882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.133392    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:19.133882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.133447    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:19.337750 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.337715    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"]
Nov 03 11:04:19.338622 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:04:19.338599    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb2139c_311d_4d53_bebf_cf7c29dd567e.slice/crio-2b9c6f9bbdacacc47a5f5ec06dbf8b8af99b6f0df043294e7cba14bfc031f348 WatchSource:0}: Error finding container 2b9c6f9bbdacacc47a5f5ec06dbf8b8af99b6f0df043294e7cba14bfc031f348: Status 404 returned error can't find the container with id 2b9c6f9bbdacacc47a5f5ec06dbf8b8af99b6f0df043294e7cba14bfc031f348
Nov 03 11:04:19.341499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.341468    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:19.341595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.341533    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:19.341595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:19.341570    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:04:20.139440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:20.139399    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" event={"ID":"ae47eae1-5e93-49ab-b252-7e5c30dd1d23","Type":"ContainerStarted","Data":"9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89"}
Nov 03 11:04:20.141240 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:20.141214    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" event={"ID":"2bb2139c-311d-4d53-bebf-cf7c29dd567e","Type":"ContainerStarted","Data":"17805146d7a56bbbef61c7748e8442fd3149a6f7e94b931a9666291d110b6f7d"}
Nov 03 11:04:20.141381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:20.141260    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" event={"ID":"2bb2139c-311d-4d53-bebf-cf7c29dd567e","Type":"ContainerStarted","Data":"2b9c6f9bbdacacc47a5f5ec06dbf8b8af99b6f0df043294e7cba14bfc031f348"}
Nov 03 11:04:20.212039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:20.211871    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podStartSLOduration=2.211846907 podStartE2EDuration="2.211846907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:20.179745969 +0000 UTC m=+2104.084600409" watchObservedRunningTime="2026-11-03 11:04:20.211846907 +0000 UTC m=+2104.116701350"
Nov 03 11:04:21.009944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.009640    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:21.039942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.039072    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:04:21.039942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.039138    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 03 11:04:21.135947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.133563    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:04:21.152987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.146061    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 11:04:21.152987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.146139    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 11:04:21.364046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.364016    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:04:21.956135 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.949856    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"]
Nov 03 11:04:21.960078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.960050    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:21.971077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.971050    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"]
Nov 03 11:04:21.981942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.973240    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"routing-overlay\""
Nov 03 11:04:22.098941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.996637    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"kube-root-ca.crt\""
Nov 03 11:04:22.098941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.996715    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d9f2ce\""
Nov 03 11:04:22.098941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.996756    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"openshift-service-ca.crt\""
Nov 03 11:04:22.098941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:21.997336    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:04:22.099315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.099006    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 03 11:04:22.099315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.011332    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"agc-cred-4909c6868a9d\""
Nov 03 11:04:22.099315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.011266    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"payload-processing-external-model-e2e--f843bc49-dockercfg-lbvq4\""
Nov 03 11:04:22.134495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.134444    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 11:04:22.135216 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.135182    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 11:04:22.187951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.160655    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-kube-api-access-xhcql\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.187951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.161064    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.187951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.161107    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.187951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.161143    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.284992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.282673    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.284992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.282726    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.284992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.282759    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.284992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.282863    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-kube-api-access-xhcql\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.284992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.284149    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.376881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.376640    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.379649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.379599    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.438068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.404656    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-kube-api-access-xhcql\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:22.673191 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:22.673154    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:23.025005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:23.015490    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:04:23.025005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:23.015559    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 03 11:04:23.149971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:23.133545    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 11:04:23.149971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:23.133606    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 11:04:23.171015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:23.162667    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"]
Nov 03 11:04:23.341657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:23.341621    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc" event={"ID":"09deaa58-dd86-4ab4-bd64-422eb54af39c","Type":"ContainerStarted","Data":"4379781c7138914b804fb89a879c12016755b8f908b8a1c167b0010b2222d497"}
Nov 03 11:04:23.993945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:23.993320    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:04:23.993945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:23.993380    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 03 11:04:24.131360 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.131325    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 11:04:24.131663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.131633    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 11:04:24.365123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.365087    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc" event={"ID":"09deaa58-dd86-4ab4-bd64-422eb54af39c","Type":"ContainerStarted","Data":"0cfb877750a65f5415d9dfb3ef3360df29f480959502108715c584debbc6cdb3"}
Nov 03 11:04:24.365561 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.365306    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:24.864623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.864565    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc" podStartSLOduration=3.864544034 podStartE2EDuration="3.864544034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:24.428036527 +0000 UTC m=+2108.332890972" watchObservedRunningTime="2026-11-03 11:04:24.864544034 +0000 UTC m=+2108.769398476"
Nov 03 11:04:24.866310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.866287    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"]
Nov 03 11:04:24.951214 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.951163    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"]
Nov 03 11:04:24.994939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.993343    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:04:24.994939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:24.993406    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 03 11:04:25.134999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:25.132310    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 11:04:25.134999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:25.132373    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 11:04:25.998060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:25.998022    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:04:25.998668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:25.998633    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 03 11:04:26.135257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:26.135049    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Nov 03 11:04:26.135257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:26.135142    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Nov 03 11:04:26.991316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:26.991224    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 03 11:04:26.991316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:26.991278    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 03 11:04:27.185851 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:27.185817    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:04:27.187470 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:27.187443    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:04:27.995547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:27.995511    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:27.996031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:27.996015    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:27.996732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:27.996706    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:04:29.866977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:29.866901    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7" podUID="65bca7ce-baf9-4911-ab93-12009db06a6d" containerName="payload-processing" containerID="cri-o://1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0" gracePeriod=25
Nov 03 11:04:29.952441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:29.952404    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b" podUID="3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" containerName="payload-processing" containerID="cri-o://928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7" gracePeriod=25
Nov 03 11:04:30.323560 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.323529    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:04:30.368624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.368586    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-plugins-config-volume\") pod \"65bca7ce-baf9-4911-ab93-12009db06a6d\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") "
Nov 03 11:04:30.368900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.368692    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-kube-api-access-zqb8p\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-kube-api-access-zqb8p\") pod \"65bca7ce-baf9-4911-ab93-12009db06a6d\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") "
Nov 03 11:04:30.368900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.368740    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-routing-overlay-volume\") pod \"65bca7ce-baf9-4911-ab93-12009db06a6d\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") "
Nov 03 11:04:30.368900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.368831    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-provider-credentials\") pod \"65bca7ce-baf9-4911-ab93-12009db06a6d\" (UID: \"65bca7ce-baf9-4911-ab93-12009db06a6d\") "
Nov 03 11:04:30.370310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.370058    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-routing-overlay-volume" pod "65bca7ce-baf9-4911-ab93-12009db06a6d" (UID: "65bca7ce-baf9-4911-ab93-12009db06a6d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:30.370310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.370277    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-plugins-config-volume" pod "65bca7ce-baf9-4911-ab93-12009db06a6d" (UID: "65bca7ce-baf9-4911-ab93-12009db06a6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:30.373176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.372676    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-provider-credentials" pod "65bca7ce-baf9-4911-ab93-12009db06a6d" (UID: "65bca7ce-baf9-4911-ab93-12009db06a6d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:30.373176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.373124    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-kube-api-access-zqb8p" pod "65bca7ce-baf9-4911-ab93-12009db06a6d" (UID: "65bca7ce-baf9-4911-ab93-12009db06a6d"). InnerVolumeSpecName "kube-api-access-zqb8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:30.400608 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.400584    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:04:30.470909 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.470822    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-plugins-config-volume\") pod \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") "
Nov 03 11:04:30.471115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.470945    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-provider-credentials\") pod \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") "
Nov 03 11:04:30.471115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.470982    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-kube-api-access-7b6g5\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-kube-api-access-7b6g5\") pod \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") "
Nov 03 11:04:30.471115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.471047    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-routing-overlay-volume\") pod \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\" (UID: \"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1\") "
Nov 03 11:04:30.471428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.471299    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-plugins-config-volume" pod "3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" (UID: "3cadeaf7-bdeb-442e-b6d8-db7b96118ea1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:30.471558 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.471476    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-routing-overlay-volume" pod "3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" (UID: "3cadeaf7-bdeb-442e-b6d8-db7b96118ea1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:30.472273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.472158    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:30.472273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.472189    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:30.472273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.472205    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:30.472273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.472220    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqb8p\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-kube-api-access-zqb8p\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:30.472273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.472236    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65bca7ce-baf9-4911-ab93-12009db06a6d-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:30.472273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.472251    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65bca7ce-baf9-4911-ab93-12009db06a6d-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:30.473728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.473704    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-kube-api-access-7b6g5" pod "3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" (UID: "3cadeaf7-bdeb-442e-b6d8-db7b96118ea1"). InnerVolumeSpecName "kube-api-access-7b6g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:30.473728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.473719    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-provider-credentials" pod "3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" (UID: "3cadeaf7-bdeb-442e-b6d8-db7b96118ea1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:30.491992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.491961    5393 generic.go:356] "Generic (PLEG): container finished" podID="3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" containerID="928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7" exitCode=0
Nov 03 11:04:30.492123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.492029    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"
Nov 03 11:04:30.492123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.492102    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b" event={"ID":"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1","Type":"ContainerDied","Data":"928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7"}
Nov 03 11:04:30.492248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.492144    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b" event={"ID":"3cadeaf7-bdeb-442e-b6d8-db7b96118ea1","Type":"ContainerDied","Data":"1fc6277d2e23ece9287ddcface6d831ccc92a4ea5e6c29b914f92e65357d8ca2"}
Nov 03 11:04:30.492248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.492171    5393 scope.go:122] "RemoveContainer" containerID="928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7"
Nov 03 11:04:30.493900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.493878    5393 generic.go:356] "Generic (PLEG): container finished" podID="65bca7ce-baf9-4911-ab93-12009db06a6d" containerID="1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0" exitCode=0
Nov 03 11:04:30.494077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.493948    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"
Nov 03 11:04:30.494077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.493958    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7" event={"ID":"65bca7ce-baf9-4911-ab93-12009db06a6d","Type":"ContainerDied","Data":"1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0"}
Nov 03 11:04:30.494077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.494032    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7" event={"ID":"65bca7ce-baf9-4911-ab93-12009db06a6d","Type":"ContainerDied","Data":"38bc3a766cdd4234f7da0f923b4184a714d60e161920ad268e21f0246a0d7832"}
Nov 03 11:04:30.506790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.506770    5393 scope.go:122] "RemoveContainer" containerID="928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7"
Nov 03 11:04:30.507104 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:04:30.507083    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7\": container with ID starting with 928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7 not found: ID does not exist" containerID="928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7"
Nov 03 11:04:30.507182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.507112    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7"} err="failed to get container status \"928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7\": rpc error: code = NotFound desc = could not find container \"928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7\": container with ID starting with 928c7b4a161b261563ff62340dcca935be7b4db3529e3c281f44df8829c614b7 not found: ID does not exist"
Nov 03 11:04:30.507182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.507128    5393 scope.go:122] "RemoveContainer" containerID="1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0"
Nov 03 11:04:30.520486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.520462    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"]
Nov 03 11:04:30.534854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.529200    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845hz67b"]
Nov 03 11:04:30.541307 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.541279    5393 scope.go:122] "RemoveContainer" containerID="1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0"
Nov 03 11:04:30.541736 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:04:30.541710    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0\": container with ID starting with 1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0 not found: ID does not exist" containerID="1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0"
Nov 03 11:04:30.541912 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.541752    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0"} err="failed to get container status \"1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0\": rpc error: code = NotFound desc = could not find container \"1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0\": container with ID starting with 1ddaec0549b07260cf5f40cb1b7d736908e1e860af8953ff9ab8ecc0938e16d0 not found: ID does not exist"
Nov 03 11:04:30.545255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.545230    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"]
Nov 03 11:04:30.550978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.550031    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65ksph7"]
Nov 03 11:04:30.573590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.573555    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:30.573590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.573587    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b6g5\" (UniqueName: \"kubernetes.io/projected/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1-kube-api-access-7b6g5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:30.609706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.609665    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"]
Nov 03 11:04:30.610512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.610490    5393 cpu_manager.go:414] "containerMap: removing container" podUID="65bca7ce-baf9-4911-ab93-12009db06a6d" containerName="payload-processing"
Nov 03 11:04:30.610512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.610512    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bca7ce-baf9-4911-ab93-12009db06a6d" containerName="payload-processing"
Nov 03 11:04:30.610698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.610587    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" containerName="payload-processing"
Nov 03 11:04:30.610698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.610596    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" containerName="payload-processing"
Nov 03 11:04:30.610804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.610790    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bca7ce-baf9-4911-ab93-12009db06a6d" containerName="payload-processing"
Nov 03 11:04:30.610862 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.610819    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" containerName="payload-processing"
Nov 03 11:04:30.615661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.615628    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.622177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.622148    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"]
Nov 03 11:04:30.625149 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.625050    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c8cb1f\"/\"openshift-service-ca.crt\""
Nov 03 11:04:30.627644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.627611    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c8cb1f\"/\"default-dockercfg-5nrd4\""
Nov 03 11:04:30.627777 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.627764    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c8cb1f\"/\"autores-model-c8cb1f-kserve-self-signed-certs\""
Nov 03 11:04:30.627848 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.627839    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c8cb1f\"/\"kube-root-ca.crt\""
Nov 03 11:04:30.674874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.674832    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af069fc8-4b3d-420d-ae26-da4648be45f3-tls-certs\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.675133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.674959    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-tmp-dir\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.675217 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.675167    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-home\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.675217 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.675213    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8sxc\" (UniqueName: \"kubernetes.io/projected/af069fc8-4b3d-420d-ae26-da4648be45f3-kube-api-access-q8sxc\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.675345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.675259    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-dshm\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.675345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.675276    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-model-cache\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.776834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.776732    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-home\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.776834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.776813    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8sxc\" (UniqueName: \"kubernetes.io/projected/af069fc8-4b3d-420d-ae26-da4648be45f3-kube-api-access-q8sxc\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.777098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.776879    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-dshm\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.777098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.776908    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-model-cache\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.777098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.777017    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af069fc8-4b3d-420d-ae26-da4648be45f3-tls-certs\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.777273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.777097    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-tmp-dir\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.777339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.777272    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-home\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.777339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.777317    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-model-cache\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.777554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.777534    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-tmp-dir\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.779661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.779630    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-dshm\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.779974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.779951    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af069fc8-4b3d-420d-ae26-da4648be45f3-tls-certs\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.798944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.786039    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8sxc\" (UniqueName: \"kubernetes.io/projected/af069fc8-4b3d-420d-ae26-da4648be45f3-kube-api-access-q8sxc\") pod \"autores-model-c8cb1f-kserve-f6cdc756f-2djxl\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") " pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:30.944585 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:30.944266    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:31.324464 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:31.324436    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"]
Nov 03 11:04:31.324964 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:04:31.324936    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf069fc8_4b3d_420d_ae26_da4648be45f3.slice/crio-93bce6d8d3a29c7dd664fbfa7e5cd56201230148a51d865e350d9065e86d311b WatchSource:0}: Error finding container 93bce6d8d3a29c7dd664fbfa7e5cd56201230148a51d865e350d9065e86d311b: Status 404 returned error can't find the container with id 93bce6d8d3a29c7dd664fbfa7e5cd56201230148a51d865e350d9065e86d311b
Nov 03 11:04:31.505596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:31.505558    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl" event={"ID":"af069fc8-4b3d-420d-ae26-da4648be45f3","Type":"ContainerStarted","Data":"a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df"}
Nov 03 11:04:31.505812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:31.505610    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl" event={"ID":"af069fc8-4b3d-420d-ae26-da4648be45f3","Type":"ContainerStarted","Data":"93bce6d8d3a29c7dd664fbfa7e5cd56201230148a51d865e350d9065e86d311b"}
Nov 03 11:04:31.525287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:31.525228    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl" podStartSLOduration=1.525213274 podStartE2EDuration="1.525213274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:31.521378181 +0000 UTC m=+2115.426232617" watchObservedRunningTime="2026-11-03 11:04:31.525213274 +0000 UTC m=+2115.430067715"
Nov 03 11:04:31.949543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:31.945560    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:32.233678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:32.233597    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cadeaf7-bdeb-442e-b6d8-db7b96118ea1" path="/var/lib/kubelet/pods/3cadeaf7-bdeb-442e-b6d8-db7b96118ea1/volumes"
Nov 03 11:04:32.234134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:32.234112    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bca7ce-baf9-4911-ab93-12009db06a6d" path="/var/lib/kubelet/pods/65bca7ce-baf9-4911-ab93-12009db06a6d/volumes"
Nov 03 11:04:35.448295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:35.448261    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:04:36.131654 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.131618    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"]
Nov 03 11:04:36.137527 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.137507    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.147114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.146824    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"]
Nov 03 11:04:36.288942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.283741    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.288942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.283892    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbq5f\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-kube-api-access-pbq5f\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.288942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.284040    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.288942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.284108    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.404994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.388121    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.404994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.388214    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.404994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.388381    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.404994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.388517    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbq5f\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-kube-api-access-pbq5f\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.404994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.398986    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.404994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.399647    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.404994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.404614    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbq5f\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-kube-api-access-pbq5f\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.461126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.461077    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") " pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:36.735371 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:36.735286    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:37.025561 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:37.025530    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"]
Nov 03 11:04:37.027275 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:04:37.027237    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e42737_5251_4cfd_a25d_bbb68239ad41.slice/crio-79728ddd255bfbdfdc5dcadb0ae0b9a4023337607bc5561267cbaf231adca10a WatchSource:0}: Error finding container 79728ddd255bfbdfdc5dcadb0ae0b9a4023337607bc5561267cbaf231adca10a: Status 404 returned error can't find the container with id 79728ddd255bfbdfdc5dcadb0ae0b9a4023337607bc5561267cbaf231adca10a
Nov 03 11:04:37.613731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:37.609932    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn" event={"ID":"c9e42737-5251-4cfd-a25d-bbb68239ad41","Type":"ContainerStarted","Data":"173fa1298a9b3e2f1733a9559af63d346c897215f53cb7c537737d5fed3a505f"}
Nov 03 11:04:37.613731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:37.610004    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn" event={"ID":"c9e42737-5251-4cfd-a25d-bbb68239ad41","Type":"ContainerStarted","Data":"79728ddd255bfbdfdc5dcadb0ae0b9a4023337607bc5561267cbaf231adca10a"}
Nov 03 11:04:37.613731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:37.611218    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:37.635015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:37.634957    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn" podStartSLOduration=1.634940021 podStartE2EDuration="1.634940021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:37.630041867 +0000 UTC m=+2121.534896309" watchObservedRunningTime="2026-11-03 11:04:37.634940021 +0000 UTC m=+2121.539794461"
Nov 03 11:04:40.944716 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:40.944674    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:40.957505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:40.957479    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:41.653516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:41.653486    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:04:43.197072 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:04:43.197037    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:04:44.468940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:44.468181    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"]
Nov 03 11:04:44.471459 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:44.471431    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:46.379800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:46.379750    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"]
Nov 03 11:04:49.468764 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:49.468716    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn" podUID="c9e42737-5251-4cfd-a25d-bbb68239ad41" containerName="payload-processing" containerID="cri-o://173fa1298a9b3e2f1733a9559af63d346c897215f53cb7c537737d5fed3a505f" gracePeriod=25
Nov 03 11:04:49.752655 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:49.752387    5393 generic.go:356] "Generic (PLEG): container finished" podID="c9e42737-5251-4cfd-a25d-bbb68239ad41" containerID="173fa1298a9b3e2f1733a9559af63d346c897215f53cb7c537737d5fed3a505f" exitCode=0
Nov 03 11:04:49.752655 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:49.752475    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn" event={"ID":"c9e42737-5251-4cfd-a25d-bbb68239ad41","Type":"ContainerDied","Data":"173fa1298a9b3e2f1733a9559af63d346c897215f53cb7c537737d5fed3a505f"}
Nov 03 11:04:50.028944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.024795    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:50.093671 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.093630    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-kube-api-access-pbq5f\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-kube-api-access-pbq5f\") pod \"c9e42737-5251-4cfd-a25d-bbb68239ad41\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") "
Nov 03 11:04:50.094470 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.094445    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-routing-overlay-volume\") pod \"c9e42737-5251-4cfd-a25d-bbb68239ad41\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") "
Nov 03 11:04:50.095125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.095105    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-plugins-config-volume\") pod \"c9e42737-5251-4cfd-a25d-bbb68239ad41\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") "
Nov 03 11:04:50.095427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.095405    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-provider-credentials\") pod \"c9e42737-5251-4cfd-a25d-bbb68239ad41\" (UID: \"c9e42737-5251-4cfd-a25d-bbb68239ad41\") "
Nov 03 11:04:50.098005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.094996    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-routing-overlay-volume" pod "c9e42737-5251-4cfd-a25d-bbb68239ad41" (UID: "c9e42737-5251-4cfd-a25d-bbb68239ad41"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:50.098159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.097967    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-plugins-config-volume" pod "c9e42737-5251-4cfd-a25d-bbb68239ad41" (UID: "c9e42737-5251-4cfd-a25d-bbb68239ad41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:04:50.103089 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.099836    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-provider-credentials" pod "c9e42737-5251-4cfd-a25d-bbb68239ad41" (UID: "c9e42737-5251-4cfd-a25d-bbb68239ad41"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:50.111214 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.111178    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-kube-api-access-pbq5f" pod "c9e42737-5251-4cfd-a25d-bbb68239ad41" (UID: "c9e42737-5251-4cfd-a25d-bbb68239ad41"). InnerVolumeSpecName "kube-api-access-pbq5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:04:50.206557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.206517    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:50.206822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.206802    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbq5f\" (UniqueName: \"kubernetes.io/projected/c9e42737-5251-4cfd-a25d-bbb68239ad41-kube-api-access-pbq5f\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:50.206968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.206953    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:50.207162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.207090    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e42737-5251-4cfd-a25d-bbb68239ad41-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:04:50.632200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.631193    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"]
Nov 03 11:04:50.632684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.632355    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c9e42737-5251-4cfd-a25d-bbb68239ad41" containerName="payload-processing"
Nov 03 11:04:50.632684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.632375    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e42737-5251-4cfd-a25d-bbb68239ad41" containerName="payload-processing"
Nov 03 11:04:50.632684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.632669    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e42737-5251-4cfd-a25d-bbb68239ad41" containerName="payload-processing"
Nov 03 11:04:50.860460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.860422    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:50.866571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.866537    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"openshift-service-ca.crt\""
Nov 03 11:04:50.866812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.866721    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"agc-cred-ad6dcb1d684f\""
Nov 03 11:04:50.866899 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.866837    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"payload-processing-external-model-e2e--8c972630-dockercfg-cvqcd\""
Nov 03 11:04:50.870735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.870695    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cbeebd\""
Nov 03 11:04:50.870885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.870810    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"routing-overlay\""
Nov 03 11:04:50.873464 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.873408    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"kube-root-ca.crt\""
Nov 03 11:04:50.908346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.908315    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"]
Nov 03 11:04:50.990253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.967059    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn" event={"ID":"c9e42737-5251-4cfd-a25d-bbb68239ad41","Type":"ContainerDied","Data":"79728ddd255bfbdfdc5dcadb0ae0b9a4023337607bc5561267cbaf231adca10a"}
Nov 03 11:04:50.990253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.967181    5393 scope.go:122] "RemoveContainer" containerID="173fa1298a9b3e2f1733a9559af63d346c897215f53cb7c537737d5fed3a505f"
Nov 03 11:04:50.990253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.967315    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"
Nov 03 11:04:50.994281 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.994244    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:50.994553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.994533    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:50.994739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.994720    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvb9\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-kube-api-access-ksvb9\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:50.994995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:50.994973    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.093976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.085323    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"]
Nov 03 11:04:51.093976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.088726    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-bdcbrr8wn"]
Nov 03 11:04:51.097677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.097635    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.097865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.097747    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.097865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.097850    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvb9\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-kube-api-access-ksvb9\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.098048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.097994    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.099107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.099078    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.099787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.099763    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.105206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.105152    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.106994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.106733    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvb9\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-kube-api-access-ksvb9\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.362881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.362847    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.650200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.650167    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"]
Nov 03 11:04:51.650619 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:04:51.650212    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac54959_444b_448c_bfea_7ea9c062c785.slice/crio-cba9375f1759ae768d083df106000ec9aa20e7d1555dd8acc6bb0af21159236a WatchSource:0}: Error finding container cba9375f1759ae768d083df106000ec9aa20e7d1555dd8acc6bb0af21159236a: Status 404 returned error can't find the container with id cba9375f1759ae768d083df106000ec9aa20e7d1555dd8acc6bb0af21159236a
Nov 03 11:04:51.977027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.976902    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc" event={"ID":"bac54959-444b-448c-bfea-7ea9c062c785","Type":"ContainerStarted","Data":"78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534"}
Nov 03 11:04:51.977027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.976993    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc" event={"ID":"bac54959-444b-448c-bfea-7ea9c062c785","Type":"ContainerStarted","Data":"cba9375f1759ae768d083df106000ec9aa20e7d1555dd8acc6bb0af21159236a"}
Nov 03 11:04:51.977280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.977058    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:04:51.999595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:51.999524    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc" podStartSLOduration=1.999504513 podStartE2EDuration="1.999504513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:04:51.991659541 +0000 UTC m=+2135.896513982" watchObservedRunningTime="2026-11-03 11:04:51.999504513 +0000 UTC m=+2135.904358956"
Nov 03 11:04:52.232569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:52.232477    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e42737-5251-4cfd-a25d-bbb68239ad41" path="/var/lib/kubelet/pods/c9e42737-5251-4cfd-a25d-bbb68239ad41/volumes"
Nov 03 11:04:56.315728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:56.315686    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"]
Nov 03 11:04:57.049288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:57.049252    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"]
Nov 03 11:04:57.062848 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:57.062802    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"]
Nov 03 11:04:57.299580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:57.298896    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"]
Nov 03 11:04:57.302704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:04:57.302493    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:05:01.316767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.316726    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7" podUID="3beccb1c-3d58-41fa-8689-f416f57e968e" containerName="payload-processing" containerID="cri-o://f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef" gracePeriod=25
Nov 03 11:05:01.411110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.411070    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl" podUID="af069fc8-4b3d-420d-ae26-da4648be45f3" containerName="main" containerID="cri-o://a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df" gracePeriod=45
Nov 03 11:05:01.698737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.698705    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:05:01.734081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.734026    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:05:01.791536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.791493    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-provider-credentials\") pod \"3beccb1c-3d58-41fa-8689-f416f57e968e\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") "
Nov 03 11:05:01.791747 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.791583    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-plugins-config-volume\") pod \"3beccb1c-3d58-41fa-8689-f416f57e968e\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") "
Nov 03 11:05:01.791830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.791748    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-kube-api-access-cj8gx\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-kube-api-access-cj8gx\") pod \"3beccb1c-3d58-41fa-8689-f416f57e968e\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") "
Nov 03 11:05:01.791900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.791855    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-routing-overlay-volume\") pod \"3beccb1c-3d58-41fa-8689-f416f57e968e\" (UID: \"3beccb1c-3d58-41fa-8689-f416f57e968e\") "
Nov 03 11:05:01.792126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.792103    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-plugins-config-volume" pod "3beccb1c-3d58-41fa-8689-f416f57e968e" (UID: "3beccb1c-3d58-41fa-8689-f416f57e968e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:01.792533 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.792508    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.792640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.792550    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-routing-overlay-volume" pod "3beccb1c-3d58-41fa-8689-f416f57e968e" (UID: "3beccb1c-3d58-41fa-8689-f416f57e968e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:01.794638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.794618    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-provider-credentials" pod "3beccb1c-3d58-41fa-8689-f416f57e968e" (UID: "3beccb1c-3d58-41fa-8689-f416f57e968e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:01.794717 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.794668    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-kube-api-access-cj8gx" pod "3beccb1c-3d58-41fa-8689-f416f57e968e" (UID: "3beccb1c-3d58-41fa-8689-f416f57e968e"). InnerVolumeSpecName "kube-api-access-cj8gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:01.893187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.893155    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-home\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-home\") pod \"af069fc8-4b3d-420d-ae26-da4648be45f3\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") "
Nov 03 11:05:01.893455 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.893432    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-home" pod "af069fc8-4b3d-420d-ae26-da4648be45f3" (UID: "af069fc8-4b3d-420d-ae26-da4648be45f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:01.893632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.893541    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af069fc8-4b3d-420d-ae26-da4648be45f3-kube-api-access-q8sxc\" (UniqueName: \"kubernetes.io/projected/af069fc8-4b3d-420d-ae26-da4648be45f3-kube-api-access-q8sxc\") pod \"af069fc8-4b3d-420d-ae26-da4648be45f3\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") "
Nov 03 11:05:01.893696 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.893628    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-model-cache\") pod \"af069fc8-4b3d-420d-ae26-da4648be45f3\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") "
Nov 03 11:05:01.893806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.893779    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-dshm\") pod \"af069fc8-4b3d-420d-ae26-da4648be45f3\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") "
Nov 03 11:05:01.893874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.893861    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-tmp-dir\") pod \"af069fc8-4b3d-420d-ae26-da4648be45f3\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") "
Nov 03 11:05:01.893955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.893935    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af069fc8-4b3d-420d-ae26-da4648be45f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/af069fc8-4b3d-420d-ae26-da4648be45f3-tls-certs\") pod \"af069fc8-4b3d-420d-ae26-da4648be45f3\" (UID: \"af069fc8-4b3d-420d-ae26-da4648be45f3\") "
Nov 03 11:05:01.895229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.894453    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-model-cache" pod "af069fc8-4b3d-420d-ae26-da4648be45f3" (UID: "af069fc8-4b3d-420d-ae26-da4648be45f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:01.895229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.894716    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8gx\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-kube-api-access-cj8gx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.895229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.894740    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3beccb1c-3d58-41fa-8689-f416f57e968e-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.895229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.894755    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.895229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.894769    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.895229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.894782    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3beccb1c-3d58-41fa-8689-f416f57e968e-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.895822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.895322    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-tmp-dir" pod "af069fc8-4b3d-420d-ae26-da4648be45f3" (UID: "af069fc8-4b3d-420d-ae26-da4648be45f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:01.899233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.899204    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af069fc8-4b3d-420d-ae26-da4648be45f3-kube-api-access-q8sxc" pod "af069fc8-4b3d-420d-ae26-da4648be45f3" (UID: "af069fc8-4b3d-420d-ae26-da4648be45f3"). InnerVolumeSpecName "kube-api-access-q8sxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:01.899507 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.899475    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af069fc8-4b3d-420d-ae26-da4648be45f3-tls-certs" pod "af069fc8-4b3d-420d-ae26-da4648be45f3" (UID: "af069fc8-4b3d-420d-ae26-da4648be45f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:05:01.899770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.899747    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-dshm" pod "af069fc8-4b3d-420d-ae26-da4648be45f3" (UID: "af069fc8-4b3d-420d-ae26-da4648be45f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:05:01.996701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.996668    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8sxc\" (UniqueName: \"kubernetes.io/projected/af069fc8-4b3d-420d-ae26-da4648be45f3-kube-api-access-q8sxc\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.996701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.996701    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.996973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.996716    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af069fc8-4b3d-420d-ae26-da4648be45f3-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:01.996973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:01.996728    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af069fc8-4b3d-420d-ae26-da4648be45f3-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:02.050460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.050419    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2" podUID="ef0b28aa-6951-4fbb-aefa-08cf5c86ae43" containerName="payload-processing" containerID="cri-o://81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95" gracePeriod=25
Nov 03 11:05:02.061520 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.061491    5393 generic.go:356] "Generic (PLEG): container finished" podID="3beccb1c-3d58-41fa-8689-f416f57e968e" containerID="f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef" exitCode=0
Nov 03 11:05:02.061680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.061532    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"
Nov 03 11:05:02.061680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.061562    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7" event={"ID":"3beccb1c-3d58-41fa-8689-f416f57e968e","Type":"ContainerDied","Data":"f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef"}
Nov 03 11:05:02.061680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.061616    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7" event={"ID":"3beccb1c-3d58-41fa-8689-f416f57e968e","Type":"ContainerDied","Data":"48fcfc498d9d126c1f4d289438f77afea092d5b65012f0f37ea409cbcc7505ce"}
Nov 03 11:05:02.061680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.061646    5393 scope.go:122] "RemoveContainer" containerID="f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef"
Nov 03 11:05:02.063032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.063009    5393 generic.go:356] "Generic (PLEG): container finished" podID="af069fc8-4b3d-420d-ae26-da4648be45f3" containerID="a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df" exitCode=0
Nov 03 11:05:02.063153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.063044    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl" event={"ID":"af069fc8-4b3d-420d-ae26-da4648be45f3","Type":"ContainerDied","Data":"a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df"}
Nov 03 11:05:02.063153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.063062    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"
Nov 03 11:05:02.063153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.063073    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl" event={"ID":"af069fc8-4b3d-420d-ae26-da4648be45f3","Type":"ContainerDied","Data":"93bce6d8d3a29c7dd664fbfa7e5cd56201230148a51d865e350d9065e86d311b"}
Nov 03 11:05:02.063153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.063111    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6" podUID="9485a2e7-7f1d-4d4f-b461-261daa51e5de" containerName="payload-pre-processing" containerID="cri-o://fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002" gracePeriod=25
Nov 03 11:05:02.183943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.183902    5393 scope.go:122] "RemoveContainer" containerID="f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef"
Nov 03 11:05:02.184442 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:05:02.184414    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef\": container with ID starting with f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef not found: ID does not exist" containerID="f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef"
Nov 03 11:05:02.184542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.184455    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef"} err="failed to get container status \"f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef\": rpc error: code = NotFound desc = could not find container \"f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef\": container with ID starting with f9b6c023e72e61ebddde7eed54c7fa5bd9e12676163e00a7ee9a74b836e231ef not found: ID does not exist"
Nov 03 11:05:02.184542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.184482    5393 scope.go:122] "RemoveContainer" containerID="a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df"
Nov 03 11:05:02.193138 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.193111    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"]
Nov 03 11:05:02.197543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.197519    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl"]
Nov 03 11:05:02.198121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.198105    5393 scope.go:122] "RemoveContainer" containerID="a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df"
Nov 03 11:05:02.198538 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:05:02.198518    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df\": container with ID starting with a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df not found: ID does not exist" containerID="a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df"
Nov 03 11:05:02.198619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.198545    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df"} err="failed to get container status \"a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df\": rpc error: code = NotFound desc = could not find container \"a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df\": container with ID starting with a7c34549c794a6a3c63da51c037f798d4cfa365cc28272b3e345e747184c83df not found: ID does not exist"
Nov 03 11:05:02.206050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.206022    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"]
Nov 03 11:05:02.211937 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.211892    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56687c/payload-processing-external-model-e2e-extmod-b-56687c-6845s89x7"]
Nov 03 11:05:02.233675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.233644    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3beccb1c-3d58-41fa-8689-f416f57e968e" path="/var/lib/kubelet/pods/3beccb1c-3d58-41fa-8689-f416f57e968e/volumes"
Nov 03 11:05:02.234221 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.234200    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af069fc8-4b3d-420d-ae26-da4648be45f3" path="/var/lib/kubelet/pods/af069fc8-4b3d-420d-ae26-da4648be45f3/volumes"
Nov 03 11:05:02.300393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.300343    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc" podUID="bac54959-444b-448c-bfea-7ea9c062c785" containerName="payload-processing" containerID="cri-o://78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534" gracePeriod=25
Nov 03 11:05:02.446573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.446541    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:05:02.451130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.451102    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:05:02.612086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.612057    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:05:02.622095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.622069    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-plugins-config-volume\") pod \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\" (UID: \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\") "
Nov 03 11:05:02.622233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.622105    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-kube-api-access-pkf4f\" (UniqueName: \"kubernetes.io/projected/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-kube-api-access-pkf4f\") pod \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\" (UID: \"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43\") "
Nov 03 11:05:02.622276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.622234    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9485a2e7-7f1d-4d4f-b461-261daa51e5de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9485a2e7-7f1d-4d4f-b461-261daa51e5de-plugins-config-volume\") pod \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\" (UID: \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\") "
Nov 03 11:05:02.622316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.622302    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9485a2e7-7f1d-4d4f-b461-261daa51e5de-kube-api-access-ntjpj\" (UniqueName: \"kubernetes.io/projected/9485a2e7-7f1d-4d4f-b461-261daa51e5de-kube-api-access-ntjpj\") pod \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\" (UID: \"9485a2e7-7f1d-4d4f-b461-261daa51e5de\") "
Nov 03 11:05:02.622555 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.622523    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-plugins-config-volume" pod "ef0b28aa-6951-4fbb-aefa-08cf5c86ae43" (UID: "ef0b28aa-6951-4fbb-aefa-08cf5c86ae43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:02.622678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.622585    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9485a2e7-7f1d-4d4f-b461-261daa51e5de-plugins-config-volume" pod "9485a2e7-7f1d-4d4f-b461-261daa51e5de" (UID: "9485a2e7-7f1d-4d4f-b461-261daa51e5de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:02.622998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.622973    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9485a2e7-7f1d-4d4f-b461-261daa51e5de-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:02.622998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.622995    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:02.624879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.624850    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-kube-api-access-pkf4f" pod "ef0b28aa-6951-4fbb-aefa-08cf5c86ae43" (UID: "ef0b28aa-6951-4fbb-aefa-08cf5c86ae43"). InnerVolumeSpecName "kube-api-access-pkf4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:02.625043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.624994    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9485a2e7-7f1d-4d4f-b461-261daa51e5de-kube-api-access-ntjpj" pod "9485a2e7-7f1d-4d4f-b461-261daa51e5de" (UID: "9485a2e7-7f1d-4d4f-b461-261daa51e5de"). InnerVolumeSpecName "kube-api-access-ntjpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:02.675071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.669939    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-c8cb1f/autores-model-c8cb1f-kserve-f6cdc756f-2djxl" podUID="af069fc8-4b3d-420d-ae26-da4648be45f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.724367    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-kube-api-access-ksvb9\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-kube-api-access-ksvb9\") pod \"bac54959-444b-448c-bfea-7ea9c062c785\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") "
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.724698    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-routing-overlay-volume\") pod \"bac54959-444b-448c-bfea-7ea9c062c785\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") "
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.724750    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-plugins-config-volume\") pod \"bac54959-444b-448c-bfea-7ea9c062c785\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") "
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.724789    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-provider-credentials\") pod \"bac54959-444b-448c-bfea-7ea9c062c785\" (UID: \"bac54959-444b-448c-bfea-7ea9c062c785\") "
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.725541    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkf4f\" (UniqueName: \"kubernetes.io/projected/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43-kube-api-access-pkf4f\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.725565    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntjpj\" (UniqueName: \"kubernetes.io/projected/9485a2e7-7f1d-4d4f-b461-261daa51e5de-kube-api-access-ntjpj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.725699    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-plugins-config-volume" pod "bac54959-444b-448c-bfea-7ea9c062c785" (UID: "bac54959-444b-448c-bfea-7ea9c062c785"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.726197    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-routing-overlay-volume" pod "bac54959-444b-448c-bfea-7ea9c062c785" (UID: "bac54959-444b-448c-bfea-7ea9c062c785"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:02.731954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.731775    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-kube-api-access-ksvb9" pod "bac54959-444b-448c-bfea-7ea9c062c785" (UID: "bac54959-444b-448c-bfea-7ea9c062c785"). InnerVolumeSpecName "kube-api-access-ksvb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:02.732580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.732282    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-provider-credentials" pod "bac54959-444b-448c-bfea-7ea9c062c785" (UID: "bac54959-444b-448c-bfea-7ea9c062c785"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:02.826875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.826835    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:02.826875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.826869    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac54959-444b-448c-bfea-7ea9c062c785-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:02.826875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.826879    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:02.827263 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:02.826888    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvb9\" (UniqueName: \"kubernetes.io/projected/bac54959-444b-448c-bfea-7ea9c062c785-kube-api-access-ksvb9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:03.073427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.073340    5393 generic.go:356] "Generic (PLEG): container finished" podID="bac54959-444b-448c-bfea-7ea9c062c785" containerID="78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534" exitCode=0
Nov 03 11:05:03.073427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.073382    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"
Nov 03 11:05:03.073427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.073416    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc" event={"ID":"bac54959-444b-448c-bfea-7ea9c062c785","Type":"ContainerDied","Data":"78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534"}
Nov 03 11:05:03.073714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.073497    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc" event={"ID":"bac54959-444b-448c-bfea-7ea9c062c785","Type":"ContainerDied","Data":"cba9375f1759ae768d083df106000ec9aa20e7d1555dd8acc6bb0af21159236a"}
Nov 03 11:05:03.073714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.073524    5393 scope.go:122] "RemoveContainer" containerID="78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534"
Nov 03 11:05:03.075148 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.075125    5393 generic.go:356] "Generic (PLEG): container finished" podID="ef0b28aa-6951-4fbb-aefa-08cf5c86ae43" containerID="81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95" exitCode=0
Nov 03 11:05:03.075428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.075262    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2" event={"ID":"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43","Type":"ContainerDied","Data":"81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95"}
Nov 03 11:05:03.075428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.075302    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2" event={"ID":"ef0b28aa-6951-4fbb-aefa-08cf5c86ae43","Type":"ContainerDied","Data":"0fbc869ece8113de8ad37a9802e873c99d811c716c003f50694f9d14e4587e04"}
Nov 03 11:05:03.075428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.075377    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"
Nov 03 11:05:03.081577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.081554    5393 generic.go:356] "Generic (PLEG): container finished" podID="9485a2e7-7f1d-4d4f-b461-261daa51e5de" containerID="fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002" exitCode=0
Nov 03 11:05:03.081986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.081678    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6" event={"ID":"9485a2e7-7f1d-4d4f-b461-261daa51e5de","Type":"ContainerDied","Data":"fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002"}
Nov 03 11:05:03.081986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.081706    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"
Nov 03 11:05:03.081986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.081745    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6" event={"ID":"9485a2e7-7f1d-4d4f-b461-261daa51e5de","Type":"ContainerDied","Data":"67441fde6b8baa64f5a3f584b593830d5236ac51a96fd4a7c320997d5d6884fe"}
Nov 03 11:05:03.088095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.088072    5393 scope.go:122] "RemoveContainer" containerID="78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534"
Nov 03 11:05:03.092157 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:05:03.092122    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534\": container with ID starting with 78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534 not found: ID does not exist" containerID="78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534"
Nov 03 11:05:03.092277 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.092170    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534"} err="failed to get container status \"78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534\": rpc error: code = NotFound desc = could not find container \"78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534\": container with ID starting with 78c1042a26f8655bf847094474460f0b249bceb8a8633ff4d1bb87b5c1b91534 not found: ID does not exist"
Nov 03 11:05:03.092277 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.092196    5393 scope.go:122] "RemoveContainer" containerID="81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95"
Nov 03 11:05:03.103085 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.103025    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"]
Nov 03 11:05:03.106695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.106675    5393 scope.go:122] "RemoveContainer" containerID="81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95"
Nov 03 11:05:03.106958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.106939    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-7dd866xvc"]
Nov 03 11:05:03.107181 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:05:03.107157    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95\": container with ID starting with 81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95 not found: ID does not exist" containerID="81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95"
Nov 03 11:05:03.107284 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.107194    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95"} err="failed to get container status \"81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95\": rpc error: code = NotFound desc = could not find container \"81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95\": container with ID starting with 81802fd1ccd641de23af28e5793f1b95ec24c744054ba0ce06fcf9b8dee0fb95 not found: ID does not exist"
Nov 03 11:05:03.107284 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.107219    5393 scope.go:122] "RemoveContainer" containerID="fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002"
Nov 03 11:05:03.116955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.116911    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"]
Nov 03 11:05:03.121043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.121013    5393 scope.go:122] "RemoveContainer" containerID="fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002"
Nov 03 11:05:03.121406 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:05:03.121380    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002\": container with ID starting with fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002 not found: ID does not exist" containerID="fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002"
Nov 03 11:05:03.121498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.121415    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002"} err="failed to get container status \"fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002\": rpc error: code = NotFound desc = could not find container \"fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002\": container with ID starting with fe95afde3bf3d8f94cdca4f9cadbae59ddf55d581428cb679e98c148779ec002 not found: ID does not exist"
Nov 03 11:05:03.121498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.121439    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c8cb1f-57bbcc4bff-6xwd6"]
Nov 03 11:05:03.135564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.135525    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"]
Nov 03 11:05:03.141104 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.141080    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c8cb1f-7b6fd79d99-dtdp2"]
Nov 03 11:05:03.665789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.665697    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"]
Nov 03 11:05:03.666568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666548    5393 cpu_manager.go:414] "containerMap: removing container" podUID="bac54959-444b-448c-bfea-7ea9c062c785" containerName="payload-processing"
Nov 03 11:05:03.666620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666570    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac54959-444b-448c-bfea-7ea9c062c785" containerName="payload-processing"
Nov 03 11:05:03.666620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666595    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3beccb1c-3d58-41fa-8689-f416f57e968e" containerName="payload-processing"
Nov 03 11:05:03.666620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666600    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3beccb1c-3d58-41fa-8689-f416f57e968e" containerName="payload-processing"
Nov 03 11:05:03.666714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666622    5393 cpu_manager.go:414] "containerMap: removing container" podUID="af069fc8-4b3d-420d-ae26-da4648be45f3" containerName="main"
Nov 03 11:05:03.666714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666627    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af069fc8-4b3d-420d-ae26-da4648be45f3" containerName="main"
Nov 03 11:05:03.666714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666634    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ef0b28aa-6951-4fbb-aefa-08cf5c86ae43" containerName="payload-processing"
Nov 03 11:05:03.666714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666639    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0b28aa-6951-4fbb-aefa-08cf5c86ae43" containerName="payload-processing"
Nov 03 11:05:03.666714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666662    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9485a2e7-7f1d-4d4f-b461-261daa51e5de" containerName="payload-pre-processing"
Nov 03 11:05:03.666714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666667    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9485a2e7-7f1d-4d4f-b461-261daa51e5de" containerName="payload-pre-processing"
Nov 03 11:05:03.666936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666790    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9485a2e7-7f1d-4d4f-b461-261daa51e5de" containerName="payload-pre-processing"
Nov 03 11:05:03.666936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666809    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac54959-444b-448c-bfea-7ea9c062c785" containerName="payload-processing"
Nov 03 11:05:03.666936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666819    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3beccb1c-3d58-41fa-8689-f416f57e968e" containerName="payload-processing"
Nov 03 11:05:03.666936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666827    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="af069fc8-4b3d-420d-ae26-da4648be45f3" containerName="main"
Nov 03 11:05:03.666936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.666833    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0b28aa-6951-4fbb-aefa-08cf5c86ae43" containerName="payload-processing"
Nov 03 11:05:03.672588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.672558    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.683356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.683278    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"]
Nov 03 11:05:03.739362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.739319    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.739583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.739396    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.739583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.739542    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.739717 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.739609    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.739717 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.739655    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.739883 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.739778    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mspj\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-kube-api-access-8mspj\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.739883 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.739841    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.740079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.739997    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3de284-0506-4d39-9a72-d5da91a31079-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.740171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.740142    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3de284-0506-4d39-9a72-d5da91a31079-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.841548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841508    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3de284-0506-4d39-9a72-d5da91a31079-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.841766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841577    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.841766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841627    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.841766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841705    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.841766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841743    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.842027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841769    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.842027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841831    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mspj\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-kube-api-access-8mspj\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.842027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841877    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.842027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.841907    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3de284-0506-4d39-9a72-d5da91a31079-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.843068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.842723    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3de284-0506-4d39-9a72-d5da91a31079-istiod-ca-cert\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.843068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.842842    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.843068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.843000    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-data\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.843309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.843118    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-certs\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.843309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.843232    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-credential-socket\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.845388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.845356    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-envoy\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.846326 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.846301    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3de284-0506-4d39-9a72-d5da91a31079-istio-podinfo\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.852468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.852447    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mspj\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-kube-api-access-8mspj\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:03.852590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:03.852549    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-istio-token\") pod \"e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") " pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:04.003100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:04.001638    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:04.232299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:04.232266    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9485a2e7-7f1d-4d4f-b461-261daa51e5de" path="/var/lib/kubelet/pods/9485a2e7-7f1d-4d4f-b461-261daa51e5de/volumes"
Nov 03 11:05:04.232731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:04.232718    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac54959-444b-448c-bfea-7ea9c062c785" path="/var/lib/kubelet/pods/bac54959-444b-448c-bfea-7ea9c062c785/volumes"
Nov 03 11:05:04.233167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:04.233146    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef0b28aa-6951-4fbb-aefa-08cf5c86ae43" path="/var/lib/kubelet/pods/ef0b28aa-6951-4fbb-aefa-08cf5c86ae43/volumes"
Nov 03 11:05:04.317210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:04.317179    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"]
Nov 03 11:05:04.317868 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:05:04.317840    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3de284_0506_4d39_9a72_d5da91a31079.slice/crio-3e00eb1b6f3b3927cd5d8a79340900063d7a3baef81138716345cbdf93feb80a WatchSource:0}: Error finding container 3e00eb1b6f3b3927cd5d8a79340900063d7a3baef81138716345cbdf93feb80a: Status 404 returned error can't find the container with id 3e00eb1b6f3b3927cd5d8a79340900063d7a3baef81138716345cbdf93feb80a
Nov 03 11:05:04.320775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:04.320745    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:04.320845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:04.320809    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:04.320845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:04.320840    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:05.131649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:05.131609    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" event={"ID":"9e3de284-0506-4d39-9a72-d5da91a31079","Type":"ContainerStarted","Data":"f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4"}
Nov 03 11:05:05.132229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:05.131674    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" event={"ID":"9e3de284-0506-4d39-9a72-d5da91a31079","Type":"ContainerStarted","Data":"3e00eb1b6f3b3927cd5d8a79340900063d7a3baef81138716345cbdf93feb80a"}
Nov 03 11:05:05.150961 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:05.150877    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" podStartSLOduration=2.150857766 podStartE2EDuration="2.150857766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:05.149446729 +0000 UTC m=+2149.054301172" watchObservedRunningTime="2026-11-03 11:05:05.150857766 +0000 UTC m=+2149.055712208"
Nov 03 11:05:05.823296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:05.823260    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"]
Nov 03 11:05:05.823567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:05.823532    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj" podUID="a6e2ee79-5674-4b90-b477-eb8635a1e51c" containerName="maas-api" containerID="cri-o://7fb43337315cb37e98250f84ab821a658d8ef197ebbf418874826f808f0bf076" gracePeriod=30
Nov 03 11:05:06.008081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.006109    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:06.008081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.006377    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 11:05:06.008081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.006424    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 11:05:06.153678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.153651    5393 generic.go:356] "Generic (PLEG): container finished" podID="a6e2ee79-5674-4b90-b477-eb8635a1e51c" containerID="7fb43337315cb37e98250f84ab821a658d8ef197ebbf418874826f808f0bf076" exitCode=0
Nov 03 11:05:06.155048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.155027    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj" event={"ID":"a6e2ee79-5674-4b90-b477-eb8635a1e51c","Type":"ContainerDied","Data":"7fb43337315cb37e98250f84ab821a658d8ef197ebbf418874826f808f0bf076"}
Nov 03 11:05:06.458777 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.457732    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:05:06.553879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.553569    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-maas-api-tls\") pod \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") "
Nov 03 11:05:06.553879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.553716    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e2ee79-5674-4b90-b477-eb8635a1e51c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e2ee79-5674-4b90-b477-eb8635a1e51c-openshift-service-ca\") pod \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") "
Nov 03 11:05:06.553879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.553775    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e2ee79-5674-4b90-b477-eb8635a1e51c-kube-api-access-fwgkx\" (UniqueName: \"kubernetes.io/projected/a6e2ee79-5674-4b90-b477-eb8635a1e51c-kube-api-access-fwgkx\") pod \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") "
Nov 03 11:05:06.553879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.553850    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-metrics-certs\") pod \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\" (UID: \"a6e2ee79-5674-4b90-b477-eb8635a1e51c\") "
Nov 03 11:05:06.567943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.561177    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e2ee79-5674-4b90-b477-eb8635a1e51c-openshift-service-ca" pod "a6e2ee79-5674-4b90-b477-eb8635a1e51c" (UID: "a6e2ee79-5674-4b90-b477-eb8635a1e51c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:05:06.607943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.605426    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-metrics-certs" pod "a6e2ee79-5674-4b90-b477-eb8635a1e51c" (UID: "a6e2ee79-5674-4b90-b477-eb8635a1e51c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:05:06.662141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.656305    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e2ee79-5674-4b90-b477-eb8635a1e51c-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:06.662141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.656351    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:06.665614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.665576    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-maas-api-tls" pod "a6e2ee79-5674-4b90-b477-eb8635a1e51c" (UID: "a6e2ee79-5674-4b90-b477-eb8635a1e51c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:05:06.666645 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.666613    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e2ee79-5674-4b90-b477-eb8635a1e51c-kube-api-access-fwgkx" pod "a6e2ee79-5674-4b90-b477-eb8635a1e51c" (UID: "a6e2ee79-5674-4b90-b477-eb8635a1e51c"). InnerVolumeSpecName "kube-api-access-fwgkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:05:06.758021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.757978    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e2ee79-5674-4b90-b477-eb8635a1e51c-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:06.758021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:06.758017    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwgkx\" (UniqueName: \"kubernetes.io/projected/a6e2ee79-5674-4b90-b477-eb8635a1e51c-kube-api-access-fwgkx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:05:07.003945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:07.003834    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 11:05:07.003945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:07.003894    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 11:05:07.163109 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:07.163079    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"
Nov 03 11:05:07.163109 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:07.163103    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj" event={"ID":"a6e2ee79-5674-4b90-b477-eb8635a1e51c","Type":"ContainerDied","Data":"7336a134671051e822c8b09212e0ecb568fbea796cab17ea6bda16c88e733c50"}
Nov 03 11:05:07.163646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:07.163171    5393 scope.go:122] "RemoveContainer" containerID="7fb43337315cb37e98250f84ab821a658d8ef197ebbf418874826f808f0bf076"
Nov 03 11:05:07.210675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:07.210634    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"]
Nov 03 11:05:07.216831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:07.216803    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c8cb1f-55979bc69f-klvxj"]
Nov 03 11:05:08.005946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:08.005343    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 11:05:08.005946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:08.005409    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 11:05:08.232098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:08.232058    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e2ee79-5674-4b90-b477-eb8635a1e51c" path="/var/lib/kubelet/pods/a6e2ee79-5674-4b90-b477-eb8635a1e51c/volumes"
Nov 03 11:05:09.006121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:09.006068    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 11:05:09.006354 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:09.006138    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 11:05:10.007946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:10.007366    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 11:05:10.007946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:10.007432    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 11:05:11.007424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:11.007380    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:11.007970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:11.007906    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:11.008614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:11.008595    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:05:13.191723 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:05:13.191680    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:05:15.735039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.734992    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"]
Nov 03 11:05:15.736134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.736108    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a6e2ee79-5674-4b90-b477-eb8635a1e51c" containerName="maas-api"
Nov 03 11:05:15.736225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.736136    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e2ee79-5674-4b90-b477-eb8635a1e51c" containerName="maas-api"
Nov 03 11:05:15.736413 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.736394    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e2ee79-5674-4b90-b477-eb8635a1e51c" containerName="maas-api"
Nov 03 11:05:15.749377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.748095    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"]
Nov 03 11:05:15.749377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.748236    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.752571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.752549    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-35936b-openshift-default-dockercfg-8mcbd\""
Nov 03 11:05:15.787838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.787798    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-podinfo\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.788207 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.788179    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-certs\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.788341 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.788231    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-credential-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.788341 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.788289    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnmt\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-kube-api-access-brnmt\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.790761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.788380    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-data\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.790761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.788485    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-envoy\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.790761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.788529    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-token\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.790761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.788742    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9fafe7-e074-44a0-953f-a53b2b47629a-istiod-ca-cert\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.790761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.788777    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894219    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-podinfo\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894331    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-certs\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894369    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-credential-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894422    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnmt\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-kube-api-access-brnmt\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894450    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-data\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894517    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-envoy\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894547    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-token\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894712    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9fafe7-e074-44a0-953f-a53b2b47629a-istiod-ca-cert\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.898336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.894742    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.902591 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.902558    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-certs\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.919258 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.903097    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-credential-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.920421 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.908270    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-envoy\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.920602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.908509    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-data\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.946941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.921412    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9fafe7-e074-44a0-953f-a53b2b47629a-istiod-ca-cert\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.947233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.925031    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-socket\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.966944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.965536    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-token\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.966944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.966140    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-podinfo\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:15.978665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:15.978597    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnmt\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-kube-api-access-brnmt\") pod \"e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") " pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:16.096996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:16.089227    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:16.631189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:16.631003    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"]
Nov 03 11:05:16.640275 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:16.640227    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:16.640401 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:16.640311    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:16.640401 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:16.640353    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:05:17.320753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:17.320644    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" event={"ID":"1e9fafe7-e074-44a0-953f-a53b2b47629a","Type":"ContainerStarted","Data":"79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b"}
Nov 03 11:05:17.320753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:17.320713    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" event={"ID":"1e9fafe7-e074-44a0-953f-a53b2b47629a","Type":"ContainerStarted","Data":"6b6c6a784672da91aa3bce3f475853203d7f4641811de5cbd1c3b46b3475ca9e"}
Nov 03 11:05:17.355654 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:17.355382    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" podStartSLOduration=2.355363171 podStartE2EDuration="2.355363171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:17.354182431 +0000 UTC m=+2161.259036873" watchObservedRunningTime="2026-11-03 11:05:17.355363171 +0000 UTC m=+2161.260217604"
Nov 03 11:05:18.095952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:18.093288    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:18.130945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:18.129205    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:18.329987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:18.329949    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:18.331058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:18.331035    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:05:20.920538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:20.920508    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"]
Nov 03 11:05:20.926823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:20.926804    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:20.932203 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:20.932161    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-35936b\""
Nov 03 11:05:20.940431 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:20.940406    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"]
Nov 03 11:05:20.957543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:20.957507    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"]
Nov 03 11:05:20.962052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:20.962030    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:20.970501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:20.970479    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-35936b-dockercfg-7zrpg\""
Nov 03 11:05:20.975820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:20.975797    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"]
Nov 03 11:05:21.088248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.088204    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2597418-4682-4239-ad90-371082e13e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q\" (UID: \"f2597418-4682-4239-ad90-371082e13e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:21.088730 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.088704    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4rr\" (UniqueName: \"kubernetes.io/projected/e2d324a2-e483-48ba-a1d5-50a5062b10b2-kube-api-access-7s4rr\") pod \"payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d\" (UID: \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:21.089209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.089173    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f882\" (UniqueName: \"kubernetes.io/projected/f2597418-4682-4239-ad90-371082e13e9b-kube-api-access-6f882\") pod \"payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q\" (UID: \"f2597418-4682-4239-ad90-371082e13e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:21.089461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.089443    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d324a2-e483-48ba-a1d5-50a5062b10b2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d\" (UID: \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:21.202993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.195611    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f882\" (UniqueName: \"kubernetes.io/projected/f2597418-4682-4239-ad90-371082e13e9b-kube-api-access-6f882\") pod \"payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q\" (UID: \"f2597418-4682-4239-ad90-371082e13e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:21.202993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.195693    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d324a2-e483-48ba-a1d5-50a5062b10b2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d\" (UID: \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:21.202993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.195795    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2597418-4682-4239-ad90-371082e13e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q\" (UID: \"f2597418-4682-4239-ad90-371082e13e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:21.202993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.195979    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4rr\" (UniqueName: \"kubernetes.io/projected/e2d324a2-e483-48ba-a1d5-50a5062b10b2-kube-api-access-7s4rr\") pod \"payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d\" (UID: \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:21.202993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.197440    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2597418-4682-4239-ad90-371082e13e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q\" (UID: \"f2597418-4682-4239-ad90-371082e13e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:21.225980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.212878    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d324a2-e483-48ba-a1d5-50a5062b10b2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d\" (UID: \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:21.225980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.217155    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4rr\" (UniqueName: \"kubernetes.io/projected/e2d324a2-e483-48ba-a1d5-50a5062b10b2-kube-api-access-7s4rr\") pod \"payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d\" (UID: \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:21.243952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.243285    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f882\" (UniqueName: \"kubernetes.io/projected/f2597418-4682-4239-ad90-371082e13e9b-kube-api-access-6f882\") pod \"payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q\" (UID: \"f2597418-4682-4239-ad90-371082e13e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:21.263957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.260190    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:21.304964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.302257    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:21.677542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.677480    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"]
Nov 03 11:05:21.682158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:21.682130    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"]
Nov 03 11:05:22.398124 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:22.398080    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d" event={"ID":"e2d324a2-e483-48ba-a1d5-50a5062b10b2","Type":"ContainerStarted","Data":"664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568"}
Nov 03 11:05:22.398574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:22.398152    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d" event={"ID":"e2d324a2-e483-48ba-a1d5-50a5062b10b2","Type":"ContainerStarted","Data":"ad5472c847c44b6250c6afde5120723e5e07b1ada994124879c182535a7c9599"}
Nov 03 11:05:22.398574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:22.398205    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:22.420710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:22.419634    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q" event={"ID":"f2597418-4682-4239-ad90-371082e13e9b","Type":"ContainerStarted","Data":"738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe"}
Nov 03 11:05:22.420710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:22.419696    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q" event={"ID":"f2597418-4682-4239-ad90-371082e13e9b","Type":"ContainerStarted","Data":"5b39b9d4d6ea4e5ed0164e24770342feace51f7a22f85869af45dcd22d7604ae"}
Nov 03 11:05:22.420710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:22.420649    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:22.486286 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:22.486230    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d" podStartSLOduration=2.486216094 podStartE2EDuration="2.486216094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:22.451642542 +0000 UTC m=+2166.356496972" watchObservedRunningTime="2026-11-03 11:05:22.486216094 +0000 UTC m=+2166.391070535"
Nov 03 11:05:22.486688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:22.486668    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q" podStartSLOduration=2.486660833 podStartE2EDuration="2.486660833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:22.482419818 +0000 UTC m=+2166.387274259" watchObservedRunningTime="2026-11-03 11:05:22.486660833 +0000 UTC m=+2166.391515273"
Nov 03 11:05:24.025799 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:24.023140    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"]
Nov 03 11:05:24.027555 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:24.027530    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-035405d5-cm67h"]
Nov 03 11:05:24.338991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:24.334216    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fff656-ef4e-4298-9753-85b6b8f31fe2" path="/var/lib/kubelet/pods/08fff656-ef4e-4298-9753-85b6b8f31fe2/volumes"
Nov 03 11:05:29.459951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.447228    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"]
Nov 03 11:05:29.467943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.463188    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.476745 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.476318    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-e2e--b8bff1be-dockercfg-45kcv\""
Nov 03 11:05:29.476745 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.476413    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-586b7631a686\""
Nov 03 11:05:29.476745 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.476600    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fb5766\""
Nov 03 11:05:29.476745 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.476647    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"routing-overlay\""
Nov 03 11:05:29.476745 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.476713    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"kube-root-ca.crt\""
Nov 03 11:05:29.494319 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.490896    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"openshift-service-ca.crt\""
Nov 03 11:05:29.557015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.556971    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.557399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.557375    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.557715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.557696    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpcd\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-kube-api-access-crpcd\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.557876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.557857    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.560995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.560969    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"]
Nov 03 11:05:29.570006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.569981    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-e40ddfc7cf41\""
Nov 03 11:05:29.666043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.664690    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpcd\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-kube-api-access-crpcd\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.666043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.664778    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.666043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.664880    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.666043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.665017    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.667228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.667200    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.668657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.668635    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.690732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.690628    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.697210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.697175    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpcd\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-kube-api-access-crpcd\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:29.873320 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:29.873288    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:30.162028 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:05:30.161995    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10a2c3d_3e8a_4e27_b3b6_586872f47b0b.slice/crio-0925bf2e32b1041f1bea249155be334cb00f8ceb457e15d93640658de1b3a5f5 WatchSource:0}: Error finding container 0925bf2e32b1041f1bea249155be334cb00f8ceb457e15d93640658de1b3a5f5: Status 404 returned error can't find the container with id 0925bf2e32b1041f1bea249155be334cb00f8ceb457e15d93640658de1b3a5f5
Nov 03 11:05:30.162641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:30.162617    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"]
Nov 03 11:05:30.578542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:30.578448    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp" event={"ID":"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b","Type":"ContainerStarted","Data":"889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01"}
Nov 03 11:05:30.578542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:30.578505    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp" event={"ID":"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b","Type":"ContainerStarted","Data":"0925bf2e32b1041f1bea249155be334cb00f8ceb457e15d93640658de1b3a5f5"}
Nov 03 11:05:30.578542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:30.578539    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:30.598559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:30.598509    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp" podStartSLOduration=1.59849448 podStartE2EDuration="1.59849448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:30.595433261 +0000 UTC m=+2174.500287715" watchObservedRunningTime="2026-11-03 11:05:30.59849448 +0000 UTC m=+2174.503348920"
Nov 03 11:05:33.025167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:33.025133    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"]
Nov 03 11:05:33.056300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:33.054482    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36e85229-kx946"]
Nov 03 11:05:33.430874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:33.430844    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:05:34.232288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:34.232244    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc23d6b3-b0e5-416d-985e-4de495d1231d" path="/var/lib/kubelet/pods/dc23d6b3-b0e5-416d-985e-4de495d1231d/volumes"
Nov 03 11:05:34.445254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:34.445225    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:05:41.590126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:41.590095    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:05:43.191682 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:05:43.191633    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:05:47.930027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:47.929982    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"]
Nov 03 11:05:47.970940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:47.970847    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:47.986021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:47.970752    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"]
Nov 03 11:05:48.017685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.017371    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.017685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.017526    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.017685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.017583    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9gv\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-kube-api-access-qj9gv\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.018259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.018235    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.208719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.198025    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.236286 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.236249    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.236588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.236564    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9gv\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-kube-api-access-qj9gv\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.241661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.236966    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.241884 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.237530    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.242092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.239604    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.249346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.249320    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.251676 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.251640    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9gv\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-kube-api-access-qj9gv\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.528612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.528525    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:48.812907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:48.812719    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"]
Nov 03 11:05:48.816532 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:05:48.816504    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3564231b_ce6e_430d_9167_9d43b10dfbab.slice/crio-ee400a49b46c66d409c33d95c1c5358a5a99904e3d24eed0c5fd951f84eb3cd7 WatchSource:0}: Error finding container ee400a49b46c66d409c33d95c1c5358a5a99904e3d24eed0c5fd951f84eb3cd7: Status 404 returned error can't find the container with id ee400a49b46c66d409c33d95c1c5358a5a99904e3d24eed0c5fd951f84eb3cd7
Nov 03 11:05:49.836054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:49.835999    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc" event={"ID":"3564231b-ce6e-430d-9167-9d43b10dfbab","Type":"ContainerStarted","Data":"6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d"}
Nov 03 11:05:49.836456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:49.836084    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc" event={"ID":"3564231b-ce6e-430d-9167-9d43b10dfbab","Type":"ContainerStarted","Data":"ee400a49b46c66d409c33d95c1c5358a5a99904e3d24eed0c5fd951f84eb3cd7"}
Nov 03 11:05:49.836456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:49.836113    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:05:49.853449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:05:49.853397    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc" podStartSLOduration=2.853382771 podStartE2EDuration="2.853382771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:05:49.851861998 +0000 UTC m=+2193.756716440" watchObservedRunningTime="2026-11-03 11:05:49.853382771 +0000 UTC m=+2193.758237212"
Nov 03 11:06:00.850533 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:00.850434    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:06:00.924981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:00.924768    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"]
Nov 03 11:06:01.455959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.455899    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"]
Nov 03 11:06:01.465498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.465462    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"]
Nov 03 11:06:01.465706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.465601    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.471556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.470956    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5711cf-openshift-default-dockercfg-fm9vx\""
Nov 03 11:06:01.551957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.548827    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpgw9\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-kube-api-access-vpgw9\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.551957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.549077    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-socket\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.598957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.591069    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-data\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.598957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.591170    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-certs\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.598957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.591232    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-credential-socket\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.598957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.591281    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-envoy\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.598957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.591306    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-podinfo\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.598957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.591376    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-token\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.598957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.591443    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06d4305-2da3-41b4-9f80-62b8934617b3-istiod-ca-cert\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.735991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.725478    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-certs\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.725032    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-certs\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.778312    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-credential-socket\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.778354    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-envoy\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.778381    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-podinfo\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.778451    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-token\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.778509    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06d4305-2da3-41b4-9f80-62b8934617b3-istiod-ca-cert\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.778563    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpgw9\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-kube-api-access-vpgw9\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.778758    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-socket\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.778886    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-data\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.779339    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-data\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.779555    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-credential-socket\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.832944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.809777    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06d4305-2da3-41b4-9f80-62b8934617b3-istiod-ca-cert\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.882556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.835590    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-socket\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.883166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.847602    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-envoy\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.883291 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.865209    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-podinfo\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.903943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.896247    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpgw9\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-kube-api-access-vpgw9\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:01.921156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:01.921087    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-token\") pod \"e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") " pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:02.102945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:02.102303    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:02.553291 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:02.553263    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"]
Nov 03 11:06:02.559016 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:06:02.558036    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06d4305_2da3_41b4_9f80_62b8934617b3.slice/crio-e2576e44969774ef01dd138650af73eed9a33e18405b119337c22ffa800e20e5 WatchSource:0}: Error finding container e2576e44969774ef01dd138650af73eed9a33e18405b119337c22ffa800e20e5: Status 404 returned error can't find the container with id e2576e44969774ef01dd138650af73eed9a33e18405b119337c22ffa800e20e5
Nov 03 11:06:02.563437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:02.563399    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:02.563641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:02.563614    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:02.563820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:02.563798    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:03.454475 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.454424    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" event={"ID":"e06d4305-2da3-41b4-9f80-62b8934617b3","Type":"ContainerStarted","Data":"a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656"}
Nov 03 11:06:03.455120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.455060    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" event={"ID":"e06d4305-2da3-41b4-9f80-62b8934617b3","Type":"ContainerStarted","Data":"e2576e44969774ef01dd138650af73eed9a33e18405b119337c22ffa800e20e5"}
Nov 03 11:06:03.714998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.648461    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"]
Nov 03 11:06:03.714998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.683593    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.734120    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqzk\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-kube-api-access-nzqzk\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.734415    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.734496    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.734541    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.735709    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"openshift-service-ca.crt\""
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.735792    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"agc-cred-b049f8315951\""
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.736057    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-e2e--b43e39c2-dockercfg-v6cdb\""
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.736772    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"routing-overlay\""
Nov 03 11:06:03.736956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.736849    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9f53bb\""
Nov 03 11:06:03.738240 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.738222    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"kube-root-ca.crt\""
Nov 03 11:06:03.781509 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.781476    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"]
Nov 03 11:06:03.800776 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.800733    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"]
Nov 03 11:06:03.827967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.826826    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:03.835708 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.833036    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d9f2ce\"/\"agc-cred-8c0d42d01a42\""
Nov 03 11:06:03.852949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.849848    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.852949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.849932    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.852949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.849986    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.852949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.850178    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqzk\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-kube-api-access-nzqzk\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.852949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.852622    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.864004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.862138    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.904945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.886156    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqzk\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-kube-api-access-nzqzk\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.904945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.904669    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:03.931867 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.927987    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"]
Nov 03 11:06:03.931867 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.931016    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:06:03.935940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.932273    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:06:03.935940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.933972    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 11:06:03.948941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.948017    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podStartSLOduration=2.947982671 podStartE2EDuration="2.947982671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:03.491545295 +0000 UTC m=+2207.396399736" watchObservedRunningTime="2026-11-03 11:06:03.947982671 +0000 UTC m=+2207.852837117"
Nov 03 11:06:03.953941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.953039    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:03.953941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.953110    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qbk\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-kube-api-access-q8qbk\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:03.953941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.953222    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:03.953941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:03.953302    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.055426 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.055319    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qbk\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-kube-api-access-q8qbk\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.055848 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.055825    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.056133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.056103    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.056501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.056479    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.057838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.057814    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.063367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.063334    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:06:04.064760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.063720    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.069761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.068999    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qbk\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-kube-api-access-q8qbk\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.069761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.069629    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") " pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.070463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.070266    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:06:04.083282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.083259    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:04.110553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.108063    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:04.110553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.109427    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 11:06:04.110553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.109495    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 11:06:04.135083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.134998    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:06:04.208736 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.208701    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:04.559951 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:06:04.559894    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13310ec7_1a25_46ec_b997_342923ac3d65.slice/crio-4e6afa2b0a19f7f3612e3f9eaade17474bdaa65141819fc5ed4fd9a0a5efa25e WatchSource:0}: Error finding container 4e6afa2b0a19f7f3612e3f9eaade17474bdaa65141819fc5ed4fd9a0a5efa25e: Status 404 returned error can't find the container with id 4e6afa2b0a19f7f3612e3f9eaade17474bdaa65141819fc5ed4fd9a0a5efa25e
Nov 03 11:06:04.566246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.566216    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"]
Nov 03 11:06:04.624072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:04.624045    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"]
Nov 03 11:06:05.117249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.117205    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:05.476647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.476563    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt" event={"ID":"13310ec7-1a25-46ec-b997-342923ac3d65","Type":"ContainerStarted","Data":"15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d"}
Nov 03 11:06:05.476647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.476618    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt" event={"ID":"13310ec7-1a25-46ec-b997-342923ac3d65","Type":"ContainerStarted","Data":"4e6afa2b0a19f7f3612e3f9eaade17474bdaa65141819fc5ed4fd9a0a5efa25e"}
Nov 03 11:06:05.476849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.476661    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:05.478683 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.478658    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b" event={"ID":"a17c31d4-0cbf-4c53-aeb4-2a32274779be","Type":"ContainerStarted","Data":"7b5ad8beb12c0056811dac80d2c5a4301a67f214b4a4318865bc370303c924f3"}
Nov 03 11:06:05.478760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.478711    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:05.478760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.478737    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b" event={"ID":"a17c31d4-0cbf-4c53-aeb4-2a32274779be","Type":"ContainerStarted","Data":"7de1f0fc3fcd62a4076f7b6e1bbd885f87451d17e3da8548742ab6652fafc24c"}
Nov 03 11:06:05.478832 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.478766    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:05.479993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.479974    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:06:05.511563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.511511    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt" podStartSLOduration=2.511496977 podStartE2EDuration="2.511496977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:05.506486548 +0000 UTC m=+2209.411341004" watchObservedRunningTime="2026-11-03 11:06:05.511496977 +0000 UTC m=+2209.416351417"
Nov 03 11:06:05.548133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.548087    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b" podStartSLOduration=2.54807059 podStartE2EDuration="2.54807059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:05.543797864 +0000 UTC m=+2209.448652305" watchObservedRunningTime="2026-11-03 11:06:05.54807059 +0000 UTC m=+2209.452925034"
Nov 03 11:06:05.926282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:05.926239    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp" podUID="b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" containerName="payload-processing" containerID="cri-o://889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01" gracePeriod=25
Nov 03 11:06:06.357945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.346215    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:06:06.421931 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.421815    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-routing-overlay-volume\") pod \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") "
Nov 03 11:06:06.421931 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.421880    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-provider-credentials\") pod \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") "
Nov 03 11:06:06.422204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.422089    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-kube-api-access-crpcd\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-kube-api-access-crpcd\") pod \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") "
Nov 03 11:06:06.422269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.422208    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-plugins-config-volume\") pod \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\" (UID: \"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b\") "
Nov 03 11:06:06.422329 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.422269    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-routing-overlay-volume" pod "b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" (UID: "b10a2c3d-3e8a-4e27-b3b6-586872f47b0b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:06.422741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.422717    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-plugins-config-volume" pod "b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" (UID: "b10a2c3d-3e8a-4e27-b3b6-586872f47b0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:06.423077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.423056    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:06.423130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.423084    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:06.424386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.424366    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-kube-api-access-crpcd" pod "b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" (UID: "b10a2c3d-3e8a-4e27-b3b6-586872f47b0b"). InnerVolumeSpecName "kube-api-access-crpcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:06.425366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.425346    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-provider-credentials" pod "b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" (UID: "b10a2c3d-3e8a-4e27-b3b6-586872f47b0b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:06.537831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.528160    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:06.537831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.528194    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crpcd\" (UniqueName: \"kubernetes.io/projected/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b-kube-api-access-crpcd\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:06.537831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.533617    5393 generic.go:356] "Generic (PLEG): container finished" podID="b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" containerID="889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01" exitCode=0
Nov 03 11:06:06.537831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.534728    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"
Nov 03 11:06:06.537831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.535689    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp" event={"ID":"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b","Type":"ContainerDied","Data":"889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01"}
Nov 03 11:06:06.537831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.535739    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp" event={"ID":"b10a2c3d-3e8a-4e27-b3b6-586872f47b0b","Type":"ContainerDied","Data":"0925bf2e32b1041f1bea249155be334cb00f8ceb457e15d93640658de1b3a5f5"}
Nov 03 11:06:06.537831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.535770    5393 scope.go:122] "RemoveContainer" containerID="889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01"
Nov 03 11:06:06.550506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.550484    5393 scope.go:122] "RemoveContainer" containerID="889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01"
Nov 03 11:06:06.550846 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:06:06.550825    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01\": container with ID starting with 889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01 not found: ID does not exist" containerID="889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01"
Nov 03 11:06:06.550901 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.550856    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01"} err="failed to get container status \"889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01\": rpc error: code = NotFound desc = could not find container \"889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01\": container with ID starting with 889cfd6ac0cc9df99e1e191c999efd7dc38dcca8472fb4a3a90b64dc73e17c01 not found: ID does not exist"
Nov 03 11:06:06.565003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.564970    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"]
Nov 03 11:06:06.568788 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:06.568763    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-dd65pmknp"]
Nov 03 11:06:08.234206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:08.234164    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" path="/var/lib/kubelet/pods/b10a2c3d-3e8a-4e27-b3b6-586872f47b0b/volumes"
Nov 03 11:06:11.579092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:11.579056    5393 generic.go:356] "Generic (PLEG): container finished" podID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerID="79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b" exitCode=0
Nov 03 11:06:11.579625 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:11.579101    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" event={"ID":"1e9fafe7-e074-44a0-953f-a53b2b47629a","Type":"ContainerDied","Data":"79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b"}
Nov 03 11:06:11.579687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:11.579638    5393 scope.go:122] "RemoveContainer" containerID="79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b"
Nov 03 11:06:11.581770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:11.581732    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:11.581858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:11.581816    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:11.581897 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:11.581857    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:12.589403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:12.589369    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" event={"ID":"1e9fafe7-e074-44a0-953f-a53b2b47629a","Type":"ContainerStarted","Data":"b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d"}
Nov 03 11:06:13.104433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:13.104341    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:06:13.106150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:13.106117    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 11:06:13.106412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:13.106386    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 11:06:13.271656 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:06:13.271618    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:06:14.100944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:14.099544    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 11:06:14.100944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:14.099603    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 11:06:15.111336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.111292    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 11:06:15.112019 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.111371    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 11:06:15.153779 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.153728    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"]
Nov 03 11:06:15.155004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.154977    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" containerName="payload-processing"
Nov 03 11:06:15.155004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.155001    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" containerName="payload-processing"
Nov 03 11:06:15.155303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.155282    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10a2c3d-3e8a-4e27-b3b6-586872f47b0b" containerName="payload-processing"
Nov 03 11:06:15.168131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.168102    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"]
Nov 03 11:06:15.168770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.168694    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:15.173141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.173105    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5711cf\""
Nov 03 11:06:15.222538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.222508    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"]
Nov 03 11:06:15.228188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.228165    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:15.229687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.229661    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"]
Nov 03 11:06:15.232605 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.232584    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5711cf-dockercfg-992pz\""
Nov 03 11:06:15.278941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.274459    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5cdx\" (UniqueName: \"kubernetes.io/projected/9d5239bc-ba6c-4d26-b548-232d9252f273-kube-api-access-w5cdx\") pod \"payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh\" (UID: \"9d5239bc-ba6c-4d26-b548-232d9252f273\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:15.278941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.274737    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5239bc-ba6c-4d26-b548-232d9252f273-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh\" (UID: \"9d5239bc-ba6c-4d26-b548-232d9252f273\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:15.278941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.274808    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c86b89-8fd8-499a-a6a5-6846dd17ac26-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn\" (UID: \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:15.278941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.274853    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8scf\" (UniqueName: \"kubernetes.io/projected/25c86b89-8fd8-499a-a6a5-6846dd17ac26-kube-api-access-v8scf\") pod \"payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn\" (UID: \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:15.395006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.381055    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5239bc-ba6c-4d26-b548-232d9252f273-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh\" (UID: \"9d5239bc-ba6c-4d26-b548-232d9252f273\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:15.395370 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.395339    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c86b89-8fd8-499a-a6a5-6846dd17ac26-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn\" (UID: \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:15.395555 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.395538    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8scf\" (UniqueName: \"kubernetes.io/projected/25c86b89-8fd8-499a-a6a5-6846dd17ac26-kube-api-access-v8scf\") pod \"payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn\" (UID: \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:15.395752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.395733    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5cdx\" (UniqueName: \"kubernetes.io/projected/9d5239bc-ba6c-4d26-b548-232d9252f273-kube-api-access-w5cdx\") pod \"payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh\" (UID: \"9d5239bc-ba6c-4d26-b548-232d9252f273\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:15.396664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.389713    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5239bc-ba6c-4d26-b548-232d9252f273-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh\" (UID: \"9d5239bc-ba6c-4d26-b548-232d9252f273\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:15.396844 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.396680    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c86b89-8fd8-499a-a6a5-6846dd17ac26-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn\" (UID: \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:15.473849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.473804    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5cdx\" (UniqueName: \"kubernetes.io/projected/9d5239bc-ba6c-4d26-b548-232d9252f273-kube-api-access-w5cdx\") pod \"payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh\" (UID: \"9d5239bc-ba6c-4d26-b548-232d9252f273\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:15.484684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.483673    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8scf\" (UniqueName: \"kubernetes.io/projected/25c86b89-8fd8-499a-a6a5-6846dd17ac26-kube-api-access-v8scf\") pod \"payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn\" (UID: \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:15.528777 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.528190    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:15.566139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.548197    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:15.916624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.916561    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"]
Nov 03 11:06:15.923643 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:06:15.923595    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5239bc_ba6c_4d26_b548_232d9252f273.slice/crio-141f843a6a1bfcaf746ea995eca0baece9177addcbe1973f39a0467a69a564b3 WatchSource:0}: Error finding container 141f843a6a1bfcaf746ea995eca0baece9177addcbe1973f39a0467a69a564b3: Status 404 returned error can't find the container with id 141f843a6a1bfcaf746ea995eca0baece9177addcbe1973f39a0467a69a564b3
Nov 03 11:06:15.926681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:15.926309    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"]
Nov 03 11:06:16.101942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.095312    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 11:06:16.101942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.095371    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 11:06:16.539271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.539242    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:16.540070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.540047    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:16.622963 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.620615    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"]
Nov 03 11:06:16.680285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.680250    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh" event={"ID":"9d5239bc-ba6c-4d26-b548-232d9252f273","Type":"ContainerStarted","Data":"69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e"}
Nov 03 11:06:16.680477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.680326    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh" event={"ID":"9d5239bc-ba6c-4d26-b548-232d9252f273","Type":"ContainerStarted","Data":"141f843a6a1bfcaf746ea995eca0baece9177addcbe1973f39a0467a69a564b3"}
Nov 03 11:06:16.680584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.680560    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:16.724772 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.724732    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn" event={"ID":"25c86b89-8fd8-499a-a6a5-6846dd17ac26","Type":"ContainerStarted","Data":"7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145"}
Nov 03 11:06:16.725015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.724799    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn" event={"ID":"25c86b89-8fd8-499a-a6a5-6846dd17ac26","Type":"ContainerStarted","Data":"548b815a72d6a24e6cdaf7b3cfb67387e52513b6c4f5cb126e96c907c08c229e"}
Nov 03 11:06:16.725015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.724829    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:16.740598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.740540    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn" podStartSLOduration=1.740527269 podStartE2EDuration="1.740527269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:16.739166489 +0000 UTC m=+2220.644020929" watchObservedRunningTime="2026-11-03 11:06:16.740527269 +0000 UTC m=+2220.645381709"
Nov 03 11:06:16.741761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:16.741720    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh" podStartSLOduration=1.741712063 podStartE2EDuration="1.741712063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:16.722153214 +0000 UTC m=+2220.627007656" watchObservedRunningTime="2026-11-03 11:06:16.741712063 +0000 UTC m=+2220.646566503"
Nov 03 11:06:17.106945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:17.106306    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 11:06:17.106945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:17.106373    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 11:06:18.102946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:18.101014    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 11:06:18.102946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:18.101078    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 11:06:18.330855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:18.330824    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:06:19.095434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:19.095404    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:06:19.097362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:19.097337    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:06:19.855989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:19.855952    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"]
Nov 03 11:06:21.634456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:21.634414    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc" podUID="09deaa58-dd86-4ab4-bd64-422eb54af39c" containerName="payload-processing" containerID="cri-o://0cfb877750a65f5415d9dfb3ef3360df29f480959502108715c584debbc6cdb3" gracePeriod=25
Nov 03 11:06:21.915626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:21.915136    5393 generic.go:356] "Generic (PLEG): container finished" podID="09deaa58-dd86-4ab4-bd64-422eb54af39c" containerID="0cfb877750a65f5415d9dfb3ef3360df29f480959502108715c584debbc6cdb3" exitCode=0
Nov 03 11:06:21.915626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:21.915466    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc" event={"ID":"09deaa58-dd86-4ab4-bd64-422eb54af39c","Type":"ContainerDied","Data":"0cfb877750a65f5415d9dfb3ef3360df29f480959502108715c584debbc6cdb3"}
Nov 03 11:06:22.396141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.396113    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:06:22.478388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.478346    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-plugins-config-volume\") pod \"09deaa58-dd86-4ab4-bd64-422eb54af39c\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") "
Nov 03 11:06:22.478564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.478406    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-routing-overlay-volume\") pod \"09deaa58-dd86-4ab4-bd64-422eb54af39c\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") "
Nov 03 11:06:22.478564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.478559    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-provider-credentials\") pod \"09deaa58-dd86-4ab4-bd64-422eb54af39c\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") "
Nov 03 11:06:22.478732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.478593    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-kube-api-access-xhcql\") pod \"09deaa58-dd86-4ab4-bd64-422eb54af39c\" (UID: \"09deaa58-dd86-4ab4-bd64-422eb54af39c\") "
Nov 03 11:06:22.478817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.478788    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-routing-overlay-volume" pod "09deaa58-dd86-4ab4-bd64-422eb54af39c" (UID: "09deaa58-dd86-4ab4-bd64-422eb54af39c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:22.478877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.478830    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-plugins-config-volume" pod "09deaa58-dd86-4ab4-bd64-422eb54af39c" (UID: "09deaa58-dd86-4ab4-bd64-422eb54af39c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:22.479166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.479146    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:22.479260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.479170    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09deaa58-dd86-4ab4-bd64-422eb54af39c-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:22.481167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.481141    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-provider-credentials" pod "09deaa58-dd86-4ab4-bd64-422eb54af39c" (UID: "09deaa58-dd86-4ab4-bd64-422eb54af39c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:22.481263 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.481178    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-kube-api-access-xhcql" pod "09deaa58-dd86-4ab4-bd64-422eb54af39c" (UID: "09deaa58-dd86-4ab4-bd64-422eb54af39c"). InnerVolumeSpecName "kube-api-access-xhcql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:22.580663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.580630    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:22.580663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.580661    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/09deaa58-dd86-4ab4-bd64-422eb54af39c-kube-api-access-xhcql\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:22.926602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.926569    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"
Nov 03 11:06:22.927409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.926599    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc" event={"ID":"09deaa58-dd86-4ab4-bd64-422eb54af39c","Type":"ContainerDied","Data":"4379781c7138914b804fb89a879c12016755b8f908b8a1c167b0010b2222d497"}
Nov 03 11:06:22.927409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.927270    5393 scope.go:122] "RemoveContainer" containerID="0cfb877750a65f5415d9dfb3ef3360df29f480959502108715c584debbc6cdb3"
Nov 03 11:06:22.973271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.973239    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"]
Nov 03 11:06:22.980201 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:22.980160    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5bd7d2zzc"]
Nov 03 11:06:24.232819 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:24.232788    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09deaa58-dd86-4ab4-bd64-422eb54af39c" path="/var/lib/kubelet/pods/09deaa58-dd86-4ab4-bd64-422eb54af39c/volumes"
Nov 03 11:06:24.858104 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:24.858067    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt" podUID="13310ec7-1a25-46ec-b997-342923ac3d65" containerName="payload-processing" containerID="cri-o://15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d" gracePeriod=25
Nov 03 11:06:25.240364 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.240334    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:25.327544 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.327506    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-plugins-config-volume\") pod \"13310ec7-1a25-46ec-b997-342923ac3d65\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") "
Nov 03 11:06:25.327742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.327620    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-provider-credentials\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-provider-credentials\") pod \"13310ec7-1a25-46ec-b997-342923ac3d65\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") "
Nov 03 11:06:25.327820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.327743    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-kube-api-access-nzqzk\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-kube-api-access-nzqzk\") pod \"13310ec7-1a25-46ec-b997-342923ac3d65\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") "
Nov 03 11:06:25.327820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.327767    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-routing-overlay-volume\") pod \"13310ec7-1a25-46ec-b997-342923ac3d65\" (UID: \"13310ec7-1a25-46ec-b997-342923ac3d65\") "
Nov 03 11:06:25.327994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.327906    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-plugins-config-volume" pod "13310ec7-1a25-46ec-b997-342923ac3d65" (UID: "13310ec7-1a25-46ec-b997-342923ac3d65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:25.328283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.328252    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-routing-overlay-volume" pod "13310ec7-1a25-46ec-b997-342923ac3d65" (UID: "13310ec7-1a25-46ec-b997-342923ac3d65"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:25.328533 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.328510    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:25.328640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.328585    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13310ec7-1a25-46ec-b997-342923ac3d65-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:25.330337 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.330317    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-provider-credentials" pod "13310ec7-1a25-46ec-b997-342923ac3d65" (UID: "13310ec7-1a25-46ec-b997-342923ac3d65"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:25.330413 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.330346    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-kube-api-access-nzqzk" pod "13310ec7-1a25-46ec-b997-342923ac3d65" (UID: "13310ec7-1a25-46ec-b997-342923ac3d65"). InnerVolumeSpecName "kube-api-access-nzqzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:25.429820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.429735    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:25.429820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.429772    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqzk\" (UniqueName: \"kubernetes.io/projected/13310ec7-1a25-46ec-b997-342923ac3d65-kube-api-access-nzqzk\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:25.964530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.964498    5393 generic.go:356] "Generic (PLEG): container finished" podID="13310ec7-1a25-46ec-b997-342923ac3d65" containerID="15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d" exitCode=0
Nov 03 11:06:25.964743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.964543    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"
Nov 03 11:06:25.964743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.964544    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt" event={"ID":"13310ec7-1a25-46ec-b997-342923ac3d65","Type":"ContainerDied","Data":"15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d"}
Nov 03 11:06:25.964743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.964719    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt" event={"ID":"13310ec7-1a25-46ec-b997-342923ac3d65","Type":"ContainerDied","Data":"4e6afa2b0a19f7f3612e3f9eaade17474bdaa65141819fc5ed4fd9a0a5efa25e"}
Nov 03 11:06:25.964939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.964758    5393 scope.go:122] "RemoveContainer" containerID="15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d"
Nov 03 11:06:25.976699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.976679    5393 scope.go:122] "RemoveContainer" containerID="15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d"
Nov 03 11:06:25.977019 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:06:25.977001    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d\": container with ID starting with 15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d not found: ID does not exist" containerID="15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d"
Nov 03 11:06:25.977083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.977032    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d"} err="failed to get container status \"15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d\": rpc error: code = NotFound desc = could not find container \"15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d\": container with ID starting with 15fe3d3810e032822232abcf78d5da1dddb7104d1a4e60d44e618e030a8e781d not found: ID does not exist"
Nov 03 11:06:25.988468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.988437    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"]
Nov 03 11:06:25.992295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:25.992274    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-59dc4ckkt"]
Nov 03 11:06:26.232536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.232461    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13310ec7-1a25-46ec-b997-342923ac3d65" path="/var/lib/kubelet/pods/13310ec7-1a25-46ec-b997-342923ac3d65/volumes"
Nov 03 11:06:26.884586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.884552    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"]
Nov 03 11:06:26.885345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.885329    5393 cpu_manager.go:414] "containerMap: removing container" podUID="09deaa58-dd86-4ab4-bd64-422eb54af39c" containerName="payload-processing"
Nov 03 11:06:26.885412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.885348    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09deaa58-dd86-4ab4-bd64-422eb54af39c" containerName="payload-processing"
Nov 03 11:06:26.885412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.885378    5393 cpu_manager.go:414] "containerMap: removing container" podUID="13310ec7-1a25-46ec-b997-342923ac3d65" containerName="payload-processing"
Nov 03 11:06:26.885412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.885384    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13310ec7-1a25-46ec-b997-342923ac3d65" containerName="payload-processing"
Nov 03 11:06:26.885518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.885500    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="13310ec7-1a25-46ec-b997-342923ac3d65" containerName="payload-processing"
Nov 03 11:06:26.885518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.885517    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="09deaa58-dd86-4ab4-bd64-422eb54af39c" containerName="payload-processing"
Nov 03 11:06:26.890734 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.890708    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.896543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.896478    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fb5766-openshift-default-dockercfg-2bbc2\""
Nov 03 11:06:26.898197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.898177    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"]
Nov 03 11:06:26.953703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.953663    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-envoy\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.953899 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.953727    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-certs\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.954196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.954159    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7266464-203d-484f-8c3c-265e27639945-istio-podinfo\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.954340 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.954239    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-istio-token\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.954340 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.954303    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.954481 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.954407    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-credential-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.954554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.954483    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-data\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.954554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.954520    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9fv\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-kube-api-access-df9fv\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:26.954554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:26.954549    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7266464-203d-484f-8c3c-265e27639945-istiod-ca-cert\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056286 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056244    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-envoy\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056311    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-certs\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056398    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7266464-203d-484f-8c3c-265e27639945-istio-podinfo\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056449    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-istio-token\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056503    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056595    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-credential-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056661    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-data\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056851 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056707    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9fv\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-kube-api-access-df9fv\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056851 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056733    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7266464-203d-484f-8c3c-265e27639945-istiod-ca-cert\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.056851 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.056774    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-certs\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.057141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.057115    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.057250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.057212    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-data\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.057405 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.057386    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-credential-socket\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.057531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.057511    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7266464-203d-484f-8c3c-265e27639945-istiod-ca-cert\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.058818 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.058797    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-envoy\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.059094 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.059078    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7266464-203d-484f-8c3c-265e27639945-istio-podinfo\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.064769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.064748    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-istio-token\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.064880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.064795    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9fv\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-kube-api-access-df9fv\") pod \"e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") " pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.209097 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.209008    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:27.572038 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:06:27.572003    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7266464_203d_484f_8c3c_265e27639945.slice/crio-2f6a3c46bf4489648284ca0c76623571bf0892baeda1491b9e85b9922c5a309f WatchSource:0}: Error finding container 2f6a3c46bf4489648284ca0c76623571bf0892baeda1491b9e85b9922c5a309f: Status 404 returned error can't find the container with id 2f6a3c46bf4489648284ca0c76623571bf0892baeda1491b9e85b9922c5a309f
Nov 03 11:06:27.572445 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.572420    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"]
Nov 03 11:06:27.575566 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.575530    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:27.575652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.575617    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:27.575691 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.575660    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:06:27.769997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.769960    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:06:27.770207 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.770049    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:06:27.868213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.868084    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"]
Nov 03 11:06:27.881562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.881526    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"]
Nov 03 11:06:27.881728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.881662    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:27.885951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.885908    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:06:27.892722 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.886466    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:06:27.997968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.982548    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jt4\" (UniqueName: \"kubernetes.io/projected/bacd3c26-3ea0-4a14-b360-deefe220755e-kube-api-access-b9jt4\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:27.997968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.982741    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:27.997968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:27.982818    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:28.043542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.043489    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" event={"ID":"f7266464-203d-484f-8c3c-265e27639945","Type":"ContainerStarted","Data":"4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412"}
Nov 03 11:06:28.043746 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.043569    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" event={"ID":"f7266464-203d-484f-8c3c-265e27639945","Type":"ContainerStarted","Data":"2f6a3c46bf4489648284ca0c76623571bf0892baeda1491b9e85b9922c5a309f"}
Nov 03 11:06:28.065767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.065713    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podStartSLOduration=2.065699581 podStartE2EDuration="2.065699581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:28.061507248 +0000 UTC m=+2231.966361689" watchObservedRunningTime="2026-11-03 11:06:28.065699581 +0000 UTC m=+2231.970554021"
Nov 03 11:06:28.094854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.094821    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:28.095013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.094885    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:28.095158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.095140    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jt4\" (UniqueName: \"kubernetes.io/projected/bacd3c26-3ea0-4a14-b360-deefe220755e-kube-api-access-b9jt4\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:28.095672 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.095646    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:28.097515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.097487    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:28.103434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.103414    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jt4\" (UniqueName: \"kubernetes.io/projected/bacd3c26-3ea0-4a14-b360-deefe220755e-kube-api-access-b9jt4\") pod \"maas-api-revoke-sub-762fcd84-fszmh\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:28.209954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.209859    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:28.211046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.211024    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:28.211441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.211420    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 11:06:28.211553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.211463    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 11:06:28.502129 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:28.502102    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"]
Nov 03 11:06:28.503283 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:06:28.503261    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacd3c26_3ea0_4a14_b360_deefe220755e.slice/crio-6bd15425108891c3fa04193e3110705a42bd0dc85b7d2e78b2909e132d30f0c4 WatchSource:0}: Error finding container 6bd15425108891c3fa04193e3110705a42bd0dc85b7d2e78b2909e132d30f0c4: Status 404 returned error can't find the container with id 6bd15425108891c3fa04193e3110705a42bd0dc85b7d2e78b2909e132d30f0c4
Nov 03 11:06:29.056646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:29.056610    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh" event={"ID":"bacd3c26-3ea0-4a14-b360-deefe220755e","Type":"ContainerStarted","Data":"da8a3f24ae87d561465f76cf3b788647445f147a4f78b15cd4c3d059b1323a83"}
Nov 03 11:06:29.057172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:29.056711    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh" event={"ID":"bacd3c26-3ea0-4a14-b360-deefe220755e","Type":"ContainerStarted","Data":"6bd15425108891c3fa04193e3110705a42bd0dc85b7d2e78b2909e132d30f0c4"}
Nov 03 11:06:29.077768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:29.077699    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh" podStartSLOduration=2.077680452 podStartE2EDuration="2.077680452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:29.072273378 +0000 UTC m=+2232.977127820" watchObservedRunningTime="2026-11-03 11:06:29.077680452 +0000 UTC m=+2232.982534895"
Nov 03 11:06:29.210209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:29.210102    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 11:06:29.210209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:29.210182    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 11:06:30.066547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:30.066455    5393 generic.go:356] "Generic (PLEG): container finished" podID="bacd3c26-3ea0-4a14-b360-deefe220755e" containerID="da8a3f24ae87d561465f76cf3b788647445f147a4f78b15cd4c3d059b1323a83" exitCode=0
Nov 03 11:06:30.066547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:30.066533    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh" event={"ID":"bacd3c26-3ea0-4a14-b360-deefe220755e","Type":"ContainerDied","Data":"da8a3f24ae87d561465f76cf3b788647445f147a4f78b15cd4c3d059b1323a83"}
Nov 03 11:06:30.210285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:30.210245    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 11:06:30.210469 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:30.210300    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 11:06:31.213378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.213347    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 11:06:31.213886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.213399    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 11:06:31.323563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.323534    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:31.463634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.463595    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-openshift-ca\") pod \"bacd3c26-3ea0-4a14-b360-deefe220755e\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") "
Nov 03 11:06:31.463843 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.463663    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-xks-ca\") pod \"bacd3c26-3ea0-4a14-b360-deefe220755e\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") "
Nov 03 11:06:31.463843 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.463779    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacd3c26-3ea0-4a14-b360-deefe220755e-kube-api-access-b9jt4\" (UniqueName: \"kubernetes.io/projected/bacd3c26-3ea0-4a14-b360-deefe220755e-kube-api-access-b9jt4\") pod \"bacd3c26-3ea0-4a14-b360-deefe220755e\" (UID: \"bacd3c26-3ea0-4a14-b360-deefe220755e\") "
Nov 03 11:06:31.464158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.464129    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-openshift-ca" pod "bacd3c26-3ea0-4a14-b360-deefe220755e" (UID: "bacd3c26-3ea0-4a14-b360-deefe220755e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:31.464393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.464378    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:31.466051 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.466028    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-xks-ca" pod "bacd3c26-3ea0-4a14-b360-deefe220755e" (UID: "bacd3c26-3ea0-4a14-b360-deefe220755e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:06:31.466126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.466105    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacd3c26-3ea0-4a14-b360-deefe220755e-kube-api-access-b9jt4" pod "bacd3c26-3ea0-4a14-b360-deefe220755e" (UID: "bacd3c26-3ea0-4a14-b360-deefe220755e"). InnerVolumeSpecName "kube-api-access-b9jt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:31.566146 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.566107    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9jt4\" (UniqueName: \"kubernetes.io/projected/bacd3c26-3ea0-4a14-b360-deefe220755e-kube-api-access-b9jt4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:31.566146 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:31.566140    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd3c26-3ea0-4a14-b360-deefe220755e-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:32.114248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:32.114212    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh" event={"ID":"bacd3c26-3ea0-4a14-b360-deefe220755e","Type":"ContainerDied","Data":"6bd15425108891c3fa04193e3110705a42bd0dc85b7d2e78b2909e132d30f0c4"}
Nov 03 11:06:32.114446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:32.114276    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd15425108891c3fa04193e3110705a42bd0dc85b7d2e78b2909e132d30f0c4"
Nov 03 11:06:32.114446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:32.114229    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"
Nov 03 11:06:32.209580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:32.209539    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 11:06:32.209775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:32.209603    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 11:06:33.211001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:33.210956    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 11:06:33.211589 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:33.211555    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 11:06:33.846321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:33.824238    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"]
Nov 03 11:06:34.211756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:34.211662    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 11:06:34.211756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:34.211724    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 11:06:35.242185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:35.242149    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:35.242976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:35.242955    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:35.243764 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:35.243748    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:06:38.854234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:38.854156    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b" podUID="a17c31d4-0cbf-4c53-aeb4-2a32274779be" containerName="payload-processing" containerID="cri-o://7b5ad8beb12c0056811dac80d2c5a4301a67f214b4a4318865bc370303c924f3" gracePeriod=25
Nov 03 11:06:38.958743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:38.958455    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"]
Nov 03 11:06:39.463269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.462244    5393 generic.go:356] "Generic (PLEG): container finished" podID="a17c31d4-0cbf-4c53-aeb4-2a32274779be" containerID="7b5ad8beb12c0056811dac80d2c5a4301a67f214b4a4318865bc370303c924f3" exitCode=0
Nov 03 11:06:39.463269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.462538    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b" event={"ID":"a17c31d4-0cbf-4c53-aeb4-2a32274779be","Type":"ContainerDied","Data":"7b5ad8beb12c0056811dac80d2c5a4301a67f214b4a4318865bc370303c924f3"}
Nov 03 11:06:39.510723 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.510420    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:39.654044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.654009    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-provider-credentials\") pod \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") "
Nov 03 11:06:39.654252 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.654066    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-routing-overlay-volume\") pod \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") "
Nov 03 11:06:39.654252 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.654165    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-plugins-config-volume\") pod \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") "
Nov 03 11:06:39.654390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.654374    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-kube-api-access-q8qbk\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-kube-api-access-q8qbk\") pod \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\" (UID: \"a17c31d4-0cbf-4c53-aeb4-2a32274779be\") "
Nov 03 11:06:39.654464 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.654436    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-routing-overlay-volume" pod "a17c31d4-0cbf-4c53-aeb4-2a32274779be" (UID: "a17c31d4-0cbf-4c53-aeb4-2a32274779be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:39.654585 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.654559    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-plugins-config-volume" pod "a17c31d4-0cbf-4c53-aeb4-2a32274779be" (UID: "a17c31d4-0cbf-4c53-aeb4-2a32274779be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:39.654910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.654890    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:39.654910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.654909    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c31d4-0cbf-4c53-aeb4-2a32274779be-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:39.656609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.656583    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-kube-api-access-q8qbk" pod "a17c31d4-0cbf-4c53-aeb4-2a32274779be" (UID: "a17c31d4-0cbf-4c53-aeb4-2a32274779be"). InnerVolumeSpecName "kube-api-access-q8qbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:39.656609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.656600    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-provider-credentials" pod "a17c31d4-0cbf-4c53-aeb4-2a32274779be" (UID: "a17c31d4-0cbf-4c53-aeb4-2a32274779be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:39.756351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.756270    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8qbk\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-kube-api-access-q8qbk\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:39.756351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:39.756300    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a17c31d4-0cbf-4c53-aeb4-2a32274779be-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:40.471799 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:40.471754    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b" event={"ID":"a17c31d4-0cbf-4c53-aeb4-2a32274779be","Type":"ContainerDied","Data":"7de1f0fc3fcd62a4076f7b6e1bbd885f87451d17e3da8548742ab6652fafc24c"}
Nov 03 11:06:40.471799 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:40.471777    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"
Nov 03 11:06:40.472382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:40.471824    5393 scope.go:122] "RemoveContainer" containerID="7b5ad8beb12c0056811dac80d2c5a4301a67f214b4a4318865bc370303c924f3"
Nov 03 11:06:40.523344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:40.523316    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"]
Nov 03 11:06:40.531592 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:40.531569    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d9f2ce/payload-processing-external-model-e2e-extmod-b-d9f2ce-5dffjwv4b"]
Nov 03 11:06:42.232644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:42.232610    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17c31d4-0cbf-4c53-aeb4-2a32274779be" path="/var/lib/kubelet/pods/a17c31d4-0cbf-4c53-aeb4-2a32274779be/volumes"
Nov 03 11:06:43.196337 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:06:43.196287    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:06:43.960005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:43.959962    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc" podUID="3564231b-ce6e-430d-9167-9d43b10dfbab" containerName="payload-processing" containerID="cri-o://6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d" gracePeriod=25
Nov 03 11:06:44.297314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.297283    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:06:44.415737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.415700    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-plugins-config-volume\") pod \"3564231b-ce6e-430d-9167-9d43b10dfbab\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") "
Nov 03 11:06:44.415737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.415752    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-kube-api-access-qj9gv\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-kube-api-access-qj9gv\") pod \"3564231b-ce6e-430d-9167-9d43b10dfbab\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") "
Nov 03 11:06:44.416056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.415830    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-provider-credentials\") pod \"3564231b-ce6e-430d-9167-9d43b10dfbab\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") "
Nov 03 11:06:44.416056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.415908    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-routing-overlay-volume\") pod \"3564231b-ce6e-430d-9167-9d43b10dfbab\" (UID: \"3564231b-ce6e-430d-9167-9d43b10dfbab\") "
Nov 03 11:06:44.416223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.416194    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-plugins-config-volume" pod "3564231b-ce6e-430d-9167-9d43b10dfbab" (UID: "3564231b-ce6e-430d-9167-9d43b10dfbab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:44.416666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.416614    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-routing-overlay-volume" pod "3564231b-ce6e-430d-9167-9d43b10dfbab" (UID: "3564231b-ce6e-430d-9167-9d43b10dfbab"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:06:44.416953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.416908    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:44.417069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.416957    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3564231b-ce6e-430d-9167-9d43b10dfbab-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:44.418449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.418421    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-provider-credentials" pod "3564231b-ce6e-430d-9167-9d43b10dfbab" (UID: "3564231b-ce6e-430d-9167-9d43b10dfbab"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:44.418550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.418515    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-kube-api-access-qj9gv" pod "3564231b-ce6e-430d-9167-9d43b10dfbab" (UID: "3564231b-ce6e-430d-9167-9d43b10dfbab"). InnerVolumeSpecName "kube-api-access-qj9gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:06:44.518662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.518561    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj9gv\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-kube-api-access-qj9gv\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:44.518662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.518589    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3564231b-ce6e-430d-9167-9d43b10dfbab-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:06:44.523521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.523489    5393 generic.go:356] "Generic (PLEG): container finished" podID="3564231b-ce6e-430d-9167-9d43b10dfbab" containerID="6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d" exitCode=0
Nov 03 11:06:44.523681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.523530    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"
Nov 03 11:06:44.523681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.523539    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc" event={"ID":"3564231b-ce6e-430d-9167-9d43b10dfbab","Type":"ContainerDied","Data":"6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d"}
Nov 03 11:06:44.523681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.523586    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc" event={"ID":"3564231b-ce6e-430d-9167-9d43b10dfbab","Type":"ContainerDied","Data":"ee400a49b46c66d409c33d95c1c5358a5a99904e3d24eed0c5fd951f84eb3cd7"}
Nov 03 11:06:44.523681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.523610    5393 scope.go:122] "RemoveContainer" containerID="6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d"
Nov 03 11:06:44.534794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.534772    5393 scope.go:122] "RemoveContainer" containerID="6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d"
Nov 03 11:06:44.535132 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:06:44.535112    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d\": container with ID starting with 6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d not found: ID does not exist" containerID="6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d"
Nov 03 11:06:44.535190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.535139    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d"} err="failed to get container status \"6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d\": rpc error: code = NotFound desc = could not find container \"6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d\": container with ID starting with 6c06299dddf63bc96677b83bf3af0711b61c660ed5266cbc75cd000ad3f86e5d not found: ID does not exist"
Nov 03 11:06:44.548168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.548143    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"]
Nov 03 11:06:44.553984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:44.553960    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-665dxfrfc"]
Nov 03 11:06:46.233962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:46.233904    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3564231b-ce6e-430d-9167-9d43b10dfbab" path="/var/lib/kubelet/pods/3564231b-ce6e-430d-9167-9d43b10dfbab/volumes"
Nov 03 11:06:48.368014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.367977    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"]
Nov 03 11:06:48.369834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.369804    5393 cpu_manager.go:414] "containerMap: removing container" podUID="bacd3c26-3ea0-4a14-b360-deefe220755e" containerName="revoke-keys"
Nov 03 11:06:48.370071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.370055    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacd3c26-3ea0-4a14-b360-deefe220755e" containerName="revoke-keys"
Nov 03 11:06:48.370220 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.370208    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a17c31d4-0cbf-4c53-aeb4-2a32274779be" containerName="payload-processing"
Nov 03 11:06:48.370315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.370304    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17c31d4-0cbf-4c53-aeb4-2a32274779be" containerName="payload-processing"
Nov 03 11:06:48.370434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.370418    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3564231b-ce6e-430d-9167-9d43b10dfbab" containerName="payload-processing"
Nov 03 11:06:48.370526 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.370515    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3564231b-ce6e-430d-9167-9d43b10dfbab" containerName="payload-processing"
Nov 03 11:06:48.382945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.370774    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacd3c26-3ea0-4a14-b360-deefe220755e" containerName="revoke-keys"
Nov 03 11:06:48.383234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.383210    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17c31d4-0cbf-4c53-aeb4-2a32274779be" containerName="payload-processing"
Nov 03 11:06:48.383357 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.383346    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3564231b-ce6e-430d-9167-9d43b10dfbab" containerName="payload-processing"
Nov 03 11:06:48.412247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.412210    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.427864 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.427819    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"]
Nov 03 11:06:48.429396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.429244    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"agc-cred-ff8b9f4705b7\""
Nov 03 11:06:48.429396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.429360    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b76c62\""
Nov 03 11:06:48.429798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.429763    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"payload-processing-external-model-e2e--0d6b7192-dockercfg-97m2g\""
Nov 03 11:06:48.468800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.468218    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"kube-root-ca.crt\""
Nov 03 11:06:48.468800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.468285    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"routing-overlay\""
Nov 03 11:06:48.468800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.468308    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b76c62\"/\"openshift-service-ca.crt\""
Nov 03 11:06:48.526399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.526348    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wzv\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-kube-api-access-p2wzv\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.526606 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.526576    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.526688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.526616    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.526688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.526651    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.628184 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.628094    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.628184 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.628147    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.628184 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.628183    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.628437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.628280    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wzv\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-kube-api-access-p2wzv\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.629111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.629080    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.629249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.629177    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.631034 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.631016    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.636074 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.636046    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wzv\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-kube-api-access-p2wzv\") pod \"payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") " pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:48.771493 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:48.771216    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:49.388819 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:49.388782    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"]
Nov 03 11:06:49.390753 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:06:49.390722    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2823f208_691c_4283_975d_f964fd2740f3.slice/crio-18e60f2fdc6a62cbb44788aeec49cf9293dee9be28b5b59958a1759bb55fbcd8 WatchSource:0}: Error finding container 18e60f2fdc6a62cbb44788aeec49cf9293dee9be28b5b59958a1759bb55fbcd8: Status 404 returned error can't find the container with id 18e60f2fdc6a62cbb44788aeec49cf9293dee9be28b5b59958a1759bb55fbcd8
Nov 03 11:06:49.632694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:49.632643    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn" event={"ID":"2823f208-691c-4283-975d-f964fd2740f3","Type":"ContainerStarted","Data":"22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33"}
Nov 03 11:06:49.632867 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:49.632716    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn" event={"ID":"2823f208-691c-4283-975d-f964fd2740f3","Type":"ContainerStarted","Data":"18e60f2fdc6a62cbb44788aeec49cf9293dee9be28b5b59958a1759bb55fbcd8"}
Nov 03 11:06:49.632867 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:49.632749    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:06:49.651750 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:49.651698    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn" podStartSLOduration=1.651682324 podStartE2EDuration="1.651682324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:49.648293169 +0000 UTC m=+2253.553147620" watchObservedRunningTime="2026-11-03 11:06:49.651682324 +0000 UTC m=+2253.556536780"
Nov 03 11:06:52.020022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.019981    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"]
Nov 03 11:06:52.033788 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.033755    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"]
Nov 03 11:06:52.035170 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.033877    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.039082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.039060    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"openshift-service-ca.crt\""
Nov 03 11:06:52.040194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.040134    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"routing-overlay\""
Nov 03 11:06:52.040543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.040275    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9f53bb\""
Nov 03 11:06:52.040543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.040350    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"kube-root-ca.crt\""
Nov 03 11:06:52.130618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.129240    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-e2e--b43e39c2-dockercfg-6rngp\""
Nov 03 11:06:52.130618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.129251    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq65g\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-kube-api-access-nq65g\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.130618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.129327    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"agc-cred-8d95d0be60f9\""
Nov 03 11:06:52.130618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.129650    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.130618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.129893    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.130618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.129964    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.237764 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.237714    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq65g\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-kube-api-access-nq65g\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.242973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.242938    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.248360 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.248331    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.254479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.254456    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.273220 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.255279    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.273454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.270550    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.273571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.254375    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.308580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.308510    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq65g\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-kube-api-access-nq65g\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.385754 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.380451    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:52.883988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:52.883790    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"]
Nov 03 11:06:52.931675 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:06:52.931644    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ed1edf_1043_4abe_940f_532c231504f4.slice/crio-c4ede28e240fc980233bc088d1adfc330901abd1f3f9cc43d7e370eab6235ae9 WatchSource:0}: Error finding container c4ede28e240fc980233bc088d1adfc330901abd1f3f9cc43d7e370eab6235ae9: Status 404 returned error can't find the container with id c4ede28e240fc980233bc088d1adfc330901abd1f3f9cc43d7e370eab6235ae9
Nov 03 11:06:53.027574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:53.027546    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"]
Nov 03 11:06:53.031150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:53.031117    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454e3372-4jrvk"]
Nov 03 11:06:53.957227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:53.957145    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw" event={"ID":"c9ed1edf-1043-4abe-940f-532c231504f4","Type":"ContainerStarted","Data":"7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58"}
Nov 03 11:06:53.957450 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:53.957291    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw" event={"ID":"c9ed1edf-1043-4abe-940f-532c231504f4","Type":"ContainerStarted","Data":"c4ede28e240fc980233bc088d1adfc330901abd1f3f9cc43d7e370eab6235ae9"}
Nov 03 11:06:53.958541 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:53.958515    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:06:53.976620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:53.976562    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw" podStartSLOduration=2.9765473719999997 podStartE2EDuration="2.976547372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:06:53.973954204 +0000 UTC m=+2257.878808645" watchObservedRunningTime="2026-11-03 11:06:53.976547372 +0000 UTC m=+2257.881401822"
Nov 03 11:06:54.233797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:54.233697    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08cca23-1a73-4fbe-9349-4a26b53fad91" path="/var/lib/kubelet/pods/b08cca23-1a73-4fbe-9349-4a26b53fad91/volumes"
Nov 03 11:06:59.967678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:59.967646    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"]
Nov 03 11:06:59.973562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:06:59.973537    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:07:03.023786 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:03.023747    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"]
Nov 03 11:07:03.028024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:03.028000    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e2a7fe-6kvbw"]
Nov 03 11:07:04.233104 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:04.233066    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f8c0d1-345b-427e-9296-888f9491a2c7" path="/var/lib/kubelet/pods/07f8c0d1-345b-427e-9296-888f9491a2c7/volumes"
Nov 03 11:07:04.968812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:04.968768    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn" podUID="2823f208-691c-4283-975d-f964fd2740f3" containerName="payload-processing" containerID="cri-o://22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33" gracePeriod=25
Nov 03 11:07:05.273116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.273090    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:07:05.375240 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.375197    5393 generic.go:356] "Generic (PLEG): container finished" podID="2823f208-691c-4283-975d-f964fd2740f3" containerID="22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33" exitCode=0
Nov 03 11:07:05.375418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.375252    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"
Nov 03 11:07:05.375418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.375281    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn" event={"ID":"2823f208-691c-4283-975d-f964fd2740f3","Type":"ContainerDied","Data":"22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33"}
Nov 03 11:07:05.375418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.375334    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn" event={"ID":"2823f208-691c-4283-975d-f964fd2740f3","Type":"ContainerDied","Data":"18e60f2fdc6a62cbb44788aeec49cf9293dee9be28b5b59958a1759bb55fbcd8"}
Nov 03 11:07:05.375418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.375358    5393 scope.go:122] "RemoveContainer" containerID="22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33"
Nov 03 11:07:05.386657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.386636    5393 scope.go:122] "RemoveContainer" containerID="22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33"
Nov 03 11:07:05.386996 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:05.386976    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33\": container with ID starting with 22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33 not found: ID does not exist" containerID="22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33"
Nov 03 11:07:05.387093 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.387005    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33"} err="failed to get container status \"22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33\": rpc error: code = NotFound desc = could not find container \"22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33\": container with ID starting with 22b7085f03f270352a353ea299187ffc634c801be679f73b96e915c22c384c33 not found: ID does not exist"
Nov 03 11:07:05.402718 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.402691    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-kube-api-access-p2wzv\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-kube-api-access-p2wzv\") pod \"2823f208-691c-4283-975d-f964fd2740f3\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") "
Nov 03 11:07:05.402941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.402903    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-plugins-config-volume\") pod \"2823f208-691c-4283-975d-f964fd2740f3\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") "
Nov 03 11:07:05.403102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.403081    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-routing-overlay-volume\") pod \"2823f208-691c-4283-975d-f964fd2740f3\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") "
Nov 03 11:07:05.403328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.403312    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-provider-credentials\") pod \"2823f208-691c-4283-975d-f964fd2740f3\" (UID: \"2823f208-691c-4283-975d-f964fd2740f3\") "
Nov 03 11:07:05.403537 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.403359    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-plugins-config-volume" pod "2823f208-691c-4283-975d-f964fd2740f3" (UID: "2823f208-691c-4283-975d-f964fd2740f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:05.403537 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.403447    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-routing-overlay-volume" pod "2823f208-691c-4283-975d-f964fd2740f3" (UID: "2823f208-691c-4283-975d-f964fd2740f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:05.404331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.404309    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:05.404447 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.404335    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2823f208-691c-4283-975d-f964fd2740f3-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:05.405534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.405511    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-kube-api-access-p2wzv" pod "2823f208-691c-4283-975d-f964fd2740f3" (UID: "2823f208-691c-4283-975d-f964fd2740f3"). InnerVolumeSpecName "kube-api-access-p2wzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:05.406144 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.406117    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-provider-credentials" pod "2823f208-691c-4283-975d-f964fd2740f3" (UID: "2823f208-691c-4283-975d-f964fd2740f3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:05.506619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.506529    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:05.506619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.506564    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wzv\" (UniqueName: \"kubernetes.io/projected/2823f208-691c-4283-975d-f964fd2740f3-kube-api-access-p2wzv\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:05.701644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.701613    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"]
Nov 03 11:07:05.710491 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:05.710459    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b76c62/payload-processing-external-model-e2e-extmod-b-b76c62-7845kfqcn"]
Nov 03 11:07:06.131070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:06.131041    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:07:06.234765 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:06.234719    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2823f208-691c-4283-975d-f964fd2740f3" path="/var/lib/kubelet/pods/2823f208-691c-4283-975d-f964fd2740f3/volumes"
Nov 03 11:07:06.275198 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:06.275164    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"]
Nov 03 11:07:06.521111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:06.521008    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:07:06.521111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:06.521091    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:07:09.537346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:09.537309    5393 generic.go:356] "Generic (PLEG): container finished" podID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerID="a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656" exitCode=0
Nov 03 11:07:09.538282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:09.537963    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" event={"ID":"e06d4305-2da3-41b4-9f80-62b8934617b3","Type":"ContainerDied","Data":"a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656"}
Nov 03 11:07:09.539082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:09.539061    5393 scope.go:122] "RemoveContainer" containerID="a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656"
Nov 03 11:07:09.542011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:09.541960    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:09.542127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:09.542080    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:09.542177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:09.542136    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:10.548685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:10.548651    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" event={"ID":"e06d4305-2da3-41b4-9f80-62b8934617b3","Type":"ContainerStarted","Data":"96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c"}
Nov 03 11:07:11.107711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.106707    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:07:11.107711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.106865    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 11:07:11.107711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.106998    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 11:07:11.451176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.451091    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw" podUID="c9ed1edf-1043-4abe-940f-532c231504f4" containerName="payload-processing" containerID="cri-o://7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58" gracePeriod=25
Nov 03 11:07:11.763383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.763359    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:07:11.770191 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.770168    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-routing-overlay-volume\") pod \"c9ed1edf-1043-4abe-940f-532c231504f4\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") "
Nov 03 11:07:11.770338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.770224    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-kube-api-access-nq65g\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-kube-api-access-nq65g\") pod \"c9ed1edf-1043-4abe-940f-532c231504f4\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") "
Nov 03 11:07:11.770338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.770284    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-plugins-config-volume\") pod \"c9ed1edf-1043-4abe-940f-532c231504f4\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") "
Nov 03 11:07:11.770474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.770354    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-provider-credentials\") pod \"c9ed1edf-1043-4abe-940f-532c231504f4\" (UID: \"c9ed1edf-1043-4abe-940f-532c231504f4\") "
Nov 03 11:07:11.770632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.770606    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-routing-overlay-volume" pod "c9ed1edf-1043-4abe-940f-532c231504f4" (UID: "c9ed1edf-1043-4abe-940f-532c231504f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:11.770778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.770751    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-plugins-config-volume" pod "c9ed1edf-1043-4abe-940f-532c231504f4" (UID: "c9ed1edf-1043-4abe-940f-532c231504f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:11.774645 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.771641    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:11.774645 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.773094    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ed1edf-1043-4abe-940f-532c231504f4-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:11.775554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.775517    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-provider-credentials" pod "c9ed1edf-1043-4abe-940f-532c231504f4" (UID: "c9ed1edf-1043-4abe-940f-532c231504f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:11.821065 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.821024    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-kube-api-access-nq65g" pod "c9ed1edf-1043-4abe-940f-532c231504f4" (UID: "c9ed1edf-1043-4abe-940f-532c231504f4"). InnerVolumeSpecName "kube-api-access-nq65g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:11.830508 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.830454    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"]
Nov 03 11:07:11.830994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.830783    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy" containerID="cri-o://9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28" gracePeriod=30
Nov 03 11:07:11.875449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.875406    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq65g\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-kube-api-access-nq65g\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:11.875449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:11.875453    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9ed1edf-1043-4abe-940f-532c231504f4-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:12.106998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.106565    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 11:07:12.106998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.106643    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 11:07:12.565778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.565684    5393 generic.go:356] "Generic (PLEG): container finished" podID="c9ed1edf-1043-4abe-940f-532c231504f4" containerID="7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58" exitCode=0
Nov 03 11:07:12.565985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.565807    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"
Nov 03 11:07:12.565985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.565856    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw" event={"ID":"c9ed1edf-1043-4abe-940f-532c231504f4","Type":"ContainerDied","Data":"7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58"}
Nov 03 11:07:12.565985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.565942    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw" event={"ID":"c9ed1edf-1043-4abe-940f-532c231504f4","Type":"ContainerDied","Data":"c4ede28e240fc980233bc088d1adfc330901abd1f3f9cc43d7e370eab6235ae9"}
Nov 03 11:07:12.565985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.565967    5393 scope.go:122] "RemoveContainer" containerID="7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58"
Nov 03 11:07:12.577122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.577104    5393 scope.go:122] "RemoveContainer" containerID="7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58"
Nov 03 11:07:12.577436 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:12.577416    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58\": container with ID starting with 7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58 not found: ID does not exist" containerID="7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58"
Nov 03 11:07:12.577480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.577445    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58"} err="failed to get container status \"7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58\": rpc error: code = NotFound desc = could not find container \"7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58\": container with ID starting with 7f1eaeedc9a9eefa8ff9a98539369b23d553d214bd578df3df5652d232675e58 not found: ID does not exist"
Nov 03 11:07:12.608172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.608132    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"]
Nov 03 11:07:12.613747 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:12.613707    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747fjkbjw"]
Nov 03 11:07:13.106681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:13.106331    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 11:07:13.106681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:13.106409    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 11:07:13.222133 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:13.222094    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:07:14.127952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.111461    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 11:07:14.127952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.111656    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 11:07:14.822854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.822820    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ed1edf-1043-4abe-940f-532c231504f4" path="/var/lib/kubelet/pods/c9ed1edf-1043-4abe-940f-532c231504f4/volumes"
Nov 03 11:07:14.835127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.835097    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"]
Nov 03 11:07:14.874405 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.874373    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c9ed1edf-1043-4abe-940f-532c231504f4" containerName="payload-processing"
Nov 03 11:07:14.875409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.875369    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ed1edf-1043-4abe-940f-532c231504f4" containerName="payload-processing"
Nov 03 11:07:14.875668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.875652    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2823f208-691c-4283-975d-f964fd2740f3" containerName="payload-processing"
Nov 03 11:07:14.875780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.875767    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2823f208-691c-4283-975d-f964fd2740f3" containerName="payload-processing"
Nov 03 11:07:14.876160 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.876141    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ed1edf-1043-4abe-940f-532c231504f4" containerName="payload-processing"
Nov 03 11:07:14.876298 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.876285    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2823f208-691c-4283-975d-f964fd2740f3" containerName="payload-processing"
Nov 03 11:07:14.912235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.897671    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"]
Nov 03 11:07:14.912235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.897843    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:14.948441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.925846    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"payload-processing-external-model-e2e--8c972630-dockercfg-mqs2l\""
Nov 03 11:07:14.948441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.944881    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cbeebd\""
Nov 03 11:07:14.948441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.944979    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"routing-overlay\""
Nov 03 11:07:14.948441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.945064    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"kube-root-ca.crt\""
Nov 03 11:07:14.948441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:14.945131    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"openshift-service-ca.crt\""
Nov 03 11:07:15.020244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.004002    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"e2e-cmp-legacy-04c87e-provider-cred\""
Nov 03 11:07:15.020244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.006086    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:07:15.140028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.084407    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:07:15.140028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.114465    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.140028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.114513    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5v4k\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-kube-api-access-q5v4k\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.140028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.114592    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.140028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.114691    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.140028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.114987    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:07:15.166193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.166151    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 11:07:15.166495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.166450    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 11:07:15.225162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.224028    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.225162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.224101    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5v4k\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-kube-api-access-q5v4k\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.225162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.224197    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.225162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.224307    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.251469 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.251433    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.254865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.254832    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.264105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.264071    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.265627 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.265601    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5v4k\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-kube-api-access-q5v4k\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.270433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.270406    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:15.573117 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:15.573083    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"]
Nov 03 11:07:15.573602 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:07:15.573576    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cf8b4c_abb1_470d_a4ca_29abe60e99dd.slice/crio-5ef0e99664d2f013ae60eae19c13e1f95621c49c70f8a7f2a0326d4d990f4d25 WatchSource:0}: Error finding container 5ef0e99664d2f013ae60eae19c13e1f95621c49c70f8a7f2a0326d4d990f4d25: Status 404 returned error can't find the container with id 5ef0e99664d2f013ae60eae19c13e1f95621c49c70f8a7f2a0326d4d990f4d25
Nov 03 11:07:16.109977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:16.105360    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 11:07:16.109977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:16.105425    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 11:07:16.255630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:16.255593    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx" event={"ID":"72cf8b4c-abb1-470d-a4ca-29abe60e99dd","Type":"ContainerStarted","Data":"8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e"}
Nov 03 11:07:16.256134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:16.255656    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx" event={"ID":"72cf8b4c-abb1-470d-a4ca-29abe60e99dd","Type":"ContainerStarted","Data":"5ef0e99664d2f013ae60eae19c13e1f95621c49c70f8a7f2a0326d4d990f4d25"}
Nov 03 11:07:16.256134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:16.255687    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:16.274488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:16.274441    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx" podStartSLOduration=2.274425662 podStartE2EDuration="2.274425662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:07:16.271045607 +0000 UTC m=+2280.175900047" watchObservedRunningTime="2026-11-03 11:07:16.274425662 +0000 UTC m=+2280.179280130"
Nov 03 11:07:17.109904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.109876    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:07:17.110443 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.110420    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:07:17.111382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.111359    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:07:17.146970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.146903    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 11:07:17.265316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.265286    5393 generic.go:356] "Generic (PLEG): container finished" podID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerID="9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28" exitCode=0
Nov 03 11:07:17.265727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.265333    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"
Nov 03 11:07:17.265727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.265375    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" event={"ID":"5170bdd1-3790-4d0a-867a-5e23959411bd","Type":"ContainerDied","Data":"9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28"}
Nov 03 11:07:17.265727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.265434    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l" event={"ID":"5170bdd1-3790-4d0a-867a-5e23959411bd","Type":"ContainerDied","Data":"c0368b68427f7001e124831d18adef29a7f9d3dff73808d8f802c75f08616c0b"}
Nov 03 11:07:17.265727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.265459    5393 scope.go:122] "RemoveContainer" containerID="9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28"
Nov 03 11:07:17.267858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.267842    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-certs\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.267972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.267879    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-envoy\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.268085 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268069    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-data\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.268173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268097    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-credential-socket\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.268173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268128    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-socket\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.268173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268167    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-token\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-token\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.268330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268201    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-certs" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:07:17.268330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268216    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-podinfo\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.268330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268282    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5170bdd1-3790-4d0a-867a-5e23959411bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5170bdd1-3790-4d0a-867a-5e23959411bd-istiod-ca-cert\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.268500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268336    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-kube-api-access-hg6bh\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-kube-api-access-hg6bh\") pod \"5170bdd1-3790-4d0a-867a-5e23959411bd\" (UID: \"5170bdd1-3790-4d0a-867a-5e23959411bd\") "
Nov 03 11:07:17.268500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268373    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-data" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:07:17.268500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268452    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-socket" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:07:17.269054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.268893    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-credential-socket" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:07:17.269244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.269213    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.269328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.269243    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.269328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.269259    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.269328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.269273    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.269889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.269864    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5170bdd1-3790-4d0a-867a-5e23959411bd-istiod-ca-cert" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:17.270468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.270445    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-envoy" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:07:17.270684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.270664    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-token" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:17.270872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.270851    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-podinfo" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:07:17.271841 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.271815    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-kube-api-access-hg6bh" pod "5170bdd1-3790-4d0a-867a-5e23959411bd" (UID: "5170bdd1-3790-4d0a-867a-5e23959411bd"). InnerVolumeSpecName "kube-api-access-hg6bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:17.316259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.316233    5393 scope.go:122] "RemoveContainer" containerID="9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28"
Nov 03 11:07:17.316613 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:17.316591    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28\": container with ID starting with 9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28 not found: ID does not exist" containerID="9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28"
Nov 03 11:07:17.316683 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.316624    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28"} err="failed to get container status \"9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28\": rpc error: code = NotFound desc = could not find container \"9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28\": container with ID starting with 9f18fe016ac1355770037595db41f37a3f7ce668c300c06617ce2092b3275d28 not found: ID does not exist"
Nov 03 11:07:17.371769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.371690    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6bh\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-kube-api-access-hg6bh\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.371769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.371720    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.371769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.371736    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.371769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.371750    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5170bdd1-3790-4d0a-867a-5e23959411bd-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.371769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.371770    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5170bdd1-3790-4d0a-867a-5e23959411bd-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:17.592078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.592035    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"]
Nov 03 11:07:17.594873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:17.594845    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tq59l"]
Nov 03 11:07:18.232472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:18.232432    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" path="/var/lib/kubelet/pods/5170bdd1-3790-4d0a-867a-5e23959411bd/volumes"
Nov 03 11:07:19.259216 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:19.259185    5393 scope.go:122] "RemoveContainer" containerID="7dd8e0341013ba875b8ddbc4eb166c264b2aef804c81198c8906c13d9488781d"
Nov 03 11:07:19.270804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:19.270778    5393 scope.go:122] "RemoveContainer" containerID="7ad3acbe10f7f35bbcbfe08ca9976ac4cbf2685eb356322cda568771240351e2"
Nov 03 11:07:27.295409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:27.291564    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:07:29.696412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.696371    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"]
Nov 03 11:07:29.697455 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.697431    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy"
Nov 03 11:07:29.697455 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.697454    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy"
Nov 03 11:07:29.697600 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.697577    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5170bdd1-3790-4d0a-867a-5e23959411bd" containerName="istio-proxy"
Nov 03 11:07:29.704028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.703994    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.709969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.709800    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"openshift-service-ca.crt\""
Nov 03 11:07:29.709969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.709837    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"kube-root-ca.crt\""
Nov 03 11:07:29.709969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.709837    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-35936b\""
Nov 03 11:07:29.709969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.709862    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"routing-overlay\""
Nov 03 11:07:29.709969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.709884    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"agc-cred-a9d681f7500c\""
Nov 03 11:07:29.710215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.710011    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"payload-processing-external-model-e2e--ac720701-dockercfg-8vnfh\""
Nov 03 11:07:29.710215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.710030    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"]
Nov 03 11:07:29.818951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.786058    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745fm\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-kube-api-access-745fm\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.818951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.786218    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.818951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.786279    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.818951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.786466    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.889003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.888953    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.889534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.889506    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.889825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.889802    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745fm\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-kube-api-access-745fm\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.890076 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.890056    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.917987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.891130    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.919815 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.919317    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.976993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.970425    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:29.976993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:29.972514    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745fm\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-kube-api-access-745fm\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:30.023940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:30.022148    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:30.386684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:30.386645    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"]
Nov 03 11:07:30.386848 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:07:30.386736    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121bb052_463b_4d74_9c71_bcc3041291b1.slice/crio-f1b53025e65990d530835dfa80d0c7150fb9bf38729e3986290aa84e8af0ffc5 WatchSource:0}: Error finding container f1b53025e65990d530835dfa80d0c7150fb9bf38729e3986290aa84e8af0ffc5: Status 404 returned error can't find the container with id f1b53025e65990d530835dfa80d0c7150fb9bf38729e3986290aa84e8af0ffc5
Nov 03 11:07:30.573128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:30.573088    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt" event={"ID":"121bb052-463b-4d74-9c71-bcc3041291b1","Type":"ContainerStarted","Data":"08d1e115bb2895c0fc3f0d25ef34b5786491d046ddb510c404b5f64002125bcb"}
Nov 03 11:07:30.573339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:30.573157    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt" event={"ID":"121bb052-463b-4d74-9c71-bcc3041291b1","Type":"ContainerStarted","Data":"f1b53025e65990d530835dfa80d0c7150fb9bf38729e3986290aa84e8af0ffc5"}
Nov 03 11:07:30.573339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:30.573219    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:30.594244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:30.594195    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt" podStartSLOduration=1.594180156 podStartE2EDuration="1.594180156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:07:30.590761905 +0000 UTC m=+2294.495616347" watchObservedRunningTime="2026-11-03 11:07:30.594180156 +0000 UTC m=+2294.499034626"
Nov 03 11:07:31.991656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:31.991621    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"]
Nov 03 11:07:32.069186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.069150    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"]
Nov 03 11:07:32.117294 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.072077    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.164946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.142075    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"e2e-cfl-895b85-provider-cred\""
Nov 03 11:07:32.438950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.400558    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.438950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.400640    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.438950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.400821    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.438950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.400862    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpnl5\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-kube-api-access-bpnl5\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.502466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.502425    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.502759 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.502736    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.503015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.502996    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.503162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.503145    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpnl5\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-kube-api-access-bpnl5\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.503798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.503773    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"]
Nov 03 11:07:32.504016 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.504000    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"]
Nov 03 11:07:32.504973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.504949    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.505554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.505527    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.506761 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:32.503884    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-a9d681f7500c: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:32.506846 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:32.506779    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:32.506887 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:32.506858    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials podName:c8211333-95a0-40a6-a0cb-5bc700143cb8 nodeName:}" failed. No retries permitted until 2026-11-03 11:07:33.006835978 +0000 UTC m=+2296.911690417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w" (UID: "c8211333-95a0-40a6-a0cb-5bc700143cb8") : secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:32.510179 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.510161    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"]
Nov 03 11:07:32.510336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.510282    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.515588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.515565    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpnl5\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-kube-api-access-bpnl5\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:32.605388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.605347    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.605654 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.605634    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.605762 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.605671    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.605881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.605864    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmvf\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-kube-api-access-ffmvf\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.713434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.713341    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.713701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.713676    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.713947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.713910    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmvf\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-kube-api-access-ffmvf\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.714253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.714216    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.715343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.715317    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.725943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.716192    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.746617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.746582    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmvf\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-kube-api-access-ffmvf\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.747308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.747285    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:32.848949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:32.846165    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:33.094967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:33.073067    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:33.094967 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:33.073306    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-a9d681f7500c: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:33.094967 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:33.073330    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:33.094967 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:33.073393    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials podName:c8211333-95a0-40a6-a0cb-5bc700143cb8 nodeName:}" failed. No retries permitted until 2026-11-03 11:07:34.073372674 +0000 UTC m=+2297.978227095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w" (UID: "c8211333-95a0-40a6-a0cb-5bc700143cb8") : secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:33.896859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:33.896824    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"]
Nov 03 11:07:33.897347 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:07:33.897317    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e9eb33_2fb8_4241_85aa_3c600c1d197d.slice/crio-a724c4b8faaffd65c9bd788a76bf3dbdfdb3d7d423fb643e366439605b9b717e WatchSource:0}: Error finding container a724c4b8faaffd65c9bd788a76bf3dbdfdb3d7d423fb643e366439605b9b717e: Status 404 returned error can't find the container with id a724c4b8faaffd65c9bd788a76bf3dbdfdb3d7d423fb643e366439605b9b717e
Nov 03 11:07:34.088772 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:34.088736    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:34.088982 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:34.088883    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-a9d681f7500c: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:34.088982 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:34.088906    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:34.088982 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:34.088976    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials podName:c8211333-95a0-40a6-a0cb-5bc700143cb8 nodeName:}" failed. No retries permitted until 2026-11-03 11:07:36.088961776 +0000 UTC m=+2299.993816195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w" (UID: "c8211333-95a0-40a6-a0cb-5bc700143cb8") : secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:34.710151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:34.710112    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs" event={"ID":"c2e9eb33-2fb8-4241-85aa-3c600c1d197d","Type":"ContainerStarted","Data":"8141c2173a0003312a41e9030930601cb4391afbda4891f423947a12df061bf6"}
Nov 03 11:07:34.710554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:34.710177    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:34.710554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:34.710195    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs" event={"ID":"c2e9eb33-2fb8-4241-85aa-3c600c1d197d","Type":"ContainerStarted","Data":"a724c4b8faaffd65c9bd788a76bf3dbdfdb3d7d423fb643e366439605b9b717e"}
Nov 03 11:07:34.730259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:34.730202    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs" podStartSLOduration=2.730183672 podStartE2EDuration="2.730183672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:07:34.727621965 +0000 UTC m=+2298.632476405" watchObservedRunningTime="2026-11-03 11:07:34.730183672 +0000 UTC m=+2298.635038117"
Nov 03 11:07:36.139379 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:36.139336    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:36.139783 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:36.139482    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-a9d681f7500c: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:36.139783 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:36.139507    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:36.139783 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:36.139576    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials podName:c8211333-95a0-40a6-a0cb-5bc700143cb8 nodeName:}" failed. No retries permitted until 2026-11-03 11:07:40.139561775 +0000 UTC m=+2304.044416211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w" (UID: "c8211333-95a0-40a6-a0cb-5bc700143cb8") : secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:37.592975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.592900    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt" podUID="121bb052-463b-4d74-9c71-bcc3041291b1" containerName="payload-processing" containerID="cri-o://08d1e115bb2895c0fc3f0d25ef34b5786491d046ddb510c404b5f64002125bcb" gracePeriod=25
Nov 03 11:07:37.743251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.743216    5393 generic.go:356] "Generic (PLEG): container finished" podID="121bb052-463b-4d74-9c71-bcc3041291b1" containerID="08d1e115bb2895c0fc3f0d25ef34b5786491d046ddb510c404b5f64002125bcb" exitCode=0
Nov 03 11:07:37.743446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.743285    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt" event={"ID":"121bb052-463b-4d74-9c71-bcc3041291b1","Type":"ContainerDied","Data":"08d1e115bb2895c0fc3f0d25ef34b5786491d046ddb510c404b5f64002125bcb"}
Nov 03 11:07:37.881775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.881746    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:37.966450 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.966413    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-plugins-config-volume\") pod \"121bb052-463b-4d74-9c71-bcc3041291b1\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") "
Nov 03 11:07:37.966641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.966506    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-routing-overlay-volume\") pod \"121bb052-463b-4d74-9c71-bcc3041291b1\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") "
Nov 03 11:07:37.966641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.966600    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-kube-api-access-745fm\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-kube-api-access-745fm\") pod \"121bb052-463b-4d74-9c71-bcc3041291b1\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") "
Nov 03 11:07:37.966738 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.966646    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-provider-credentials\") pod \"121bb052-463b-4d74-9c71-bcc3041291b1\" (UID: \"121bb052-463b-4d74-9c71-bcc3041291b1\") "
Nov 03 11:07:37.966826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.966802    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-plugins-config-volume" pod "121bb052-463b-4d74-9c71-bcc3041291b1" (UID: "121bb052-463b-4d74-9c71-bcc3041291b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:37.966879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.966854    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-routing-overlay-volume" pod "121bb052-463b-4d74-9c71-bcc3041291b1" (UID: "121bb052-463b-4d74-9c71-bcc3041291b1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:37.967455 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.967391    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:37.967455 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.967411    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/121bb052-463b-4d74-9c71-bcc3041291b1-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:37.969134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.969110    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-kube-api-access-745fm" pod "121bb052-463b-4d74-9c71-bcc3041291b1" (UID: "121bb052-463b-4d74-9c71-bcc3041291b1"). InnerVolumeSpecName "kube-api-access-745fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:37.969230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:37.969136    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-provider-credentials" pod "121bb052-463b-4d74-9c71-bcc3041291b1" (UID: "121bb052-463b-4d74-9c71-bcc3041291b1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:38.075494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:38.075445    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:38.075736 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:38.075717    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745fm\" (UniqueName: \"kubernetes.io/projected/121bb052-463b-4d74-9c71-bcc3041291b1-kube-api-access-745fm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:38.761656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:38.761627    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"
Nov 03 11:07:38.762207 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:38.761651    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt" event={"ID":"121bb052-463b-4d74-9c71-bcc3041291b1","Type":"ContainerDied","Data":"f1b53025e65990d530835dfa80d0c7150fb9bf38729e3986290aa84e8af0ffc5"}
Nov 03 11:07:38.762207 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:38.761719    5393 scope.go:122] "RemoveContainer" containerID="08d1e115bb2895c0fc3f0d25ef34b5786491d046ddb510c404b5f64002125bcb"
Nov 03 11:07:38.781277 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:38.781243    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"]
Nov 03 11:07:38.785497 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:38.785473    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fh6mnt"]
Nov 03 11:07:40.313942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.225285    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:40.404480 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:40.231124    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-35936b/agc-cred-a9d681f7500c: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:40.404856 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:40.404831    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w: secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:40.415949 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:40.405273    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials podName:c8211333-95a0-40a6-a0cb-5bc700143cb8 nodeName:}" failed. No retries permitted until 2026-11-03 11:07:48.405248074 +0000 UTC m=+2312.310102493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w" (UID: "c8211333-95a0-40a6-a0cb-5bc700143cb8") : secret "agc-cred-a9d681f7500c" not found
Nov 03 11:07:40.416590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.403634    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121bb052-463b-4d74-9c71-bcc3041291b1" path="/var/lib/kubelet/pods/121bb052-463b-4d74-9c71-bcc3041291b1/volumes"
Nov 03 11:07:40.456596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.456393    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"]
Nov 03 11:07:40.465141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.463432    5393 cpu_manager.go:414] "containerMap: removing container" podUID="121bb052-463b-4d74-9c71-bcc3041291b1" containerName="payload-processing"
Nov 03 11:07:40.480023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.479980    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121bb052-463b-4d74-9c71-bcc3041291b1" containerName="payload-processing"
Nov 03 11:07:40.480661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.480621    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="121bb052-463b-4d74-9c71-bcc3041291b1" containerName="payload-processing"
Nov 03 11:07:40.516944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.502208    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.541029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.519774    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.541029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.519842    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86c7c\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-kube-api-access-86c7c\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.541029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.519903    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.586280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.586243    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.587163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.532468    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"kube-root-ca.crt\""
Nov 03 11:07:40.587163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.541624    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"e2e-cmp-legacy-4f11cd-provider-cred\""
Nov 03 11:07:40.587353 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.541676    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"payload-processing-external-model-plugins-e2e-extmod-a-185908\""
Nov 03 11:07:40.587353 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.541764    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"openshift-service-ca.crt\""
Nov 03 11:07:40.587353 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.571024    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"payload-processing-external-model-e2e--c9e9a1ee-dockercfg-6jn22\""
Nov 03 11:07:40.587353 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.541553    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-185908\"/\"routing-overlay\""
Nov 03 11:07:40.587559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.587352    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"]
Nov 03 11:07:40.689497 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.689450    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.689711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.689528    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86c7c\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-kube-api-access-86c7c\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.689711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.689597    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.689861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.689714    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.690500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.690429    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.690500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.690469    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.692628 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.692600    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.698788 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.698759    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86c7c\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-kube-api-access-86c7c\") pod \"payload-processing-external-model-e2e-extmod-a-185908-6955szd5c\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") " pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:40.899673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:40.899612    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:41.185611 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:41.185579    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"]
Nov 03 11:07:41.186454 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:07:41.186426    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49c2cff_90a0_4fb7_bf77_c63cc239e69f.slice/crio-228462269fe247341b9f571e253f3943cc775cb6a82e26f216de51cb67312d46 WatchSource:0}: Error finding container 228462269fe247341b9f571e253f3943cc775cb6a82e26f216de51cb67312d46: Status 404 returned error can't find the container with id 228462269fe247341b9f571e253f3943cc775cb6a82e26f216de51cb67312d46
Nov 03 11:07:41.824575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:41.824530    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c" event={"ID":"f49c2cff-90a0-4fb7-bf77-c63cc239e69f","Type":"ContainerStarted","Data":"f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92"}
Nov 03 11:07:41.824988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:41.824595    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c" event={"ID":"f49c2cff-90a0-4fb7-bf77-c63cc239e69f","Type":"ContainerStarted","Data":"228462269fe247341b9f571e253f3943cc775cb6a82e26f216de51cb67312d46"}
Nov 03 11:07:41.824988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:41.824630    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:41.847400 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:41.847344    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c" podStartSLOduration=1.8473280509999999 podStartE2EDuration="1.847328051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:07:41.842816795 +0000 UTC m=+2305.747671237" watchObservedRunningTime="2026-11-03 11:07:41.847328051 +0000 UTC m=+2305.752182544"
Nov 03 11:07:43.192122 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:43.192081    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:07:45.721485 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.721453    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:07:45.800991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.799616    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"]
Nov 03 11:07:45.800991 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:07:45.800243    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w" podUID="c8211333-95a0-40a6-a0cb-5bc700143cb8"
Nov 03 11:07:45.859688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.859657    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:45.870385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.870348    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:45.910849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.910817    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-kube-api-access-bpnl5\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-kube-api-access-bpnl5\") pod \"c8211333-95a0-40a6-a0cb-5bc700143cb8\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") "
Nov 03 11:07:45.911095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.911078    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-plugins-config-volume\") pod \"c8211333-95a0-40a6-a0cb-5bc700143cb8\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") "
Nov 03 11:07:45.911206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.911193    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-routing-overlay-volume\") pod \"c8211333-95a0-40a6-a0cb-5bc700143cb8\" (UID: \"c8211333-95a0-40a6-a0cb-5bc700143cb8\") "
Nov 03 11:07:45.911553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.911528    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-plugins-config-volume" pod "c8211333-95a0-40a6-a0cb-5bc700143cb8" (UID: "c8211333-95a0-40a6-a0cb-5bc700143cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:45.911695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.911585    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-routing-overlay-volume" pod "c8211333-95a0-40a6-a0cb-5bc700143cb8" (UID: "c8211333-95a0-40a6-a0cb-5bc700143cb8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:07:45.912033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.912009    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:45.912033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.912034    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8211333-95a0-40a6-a0cb-5bc700143cb8-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:45.913875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:45.913850    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-kube-api-access-bpnl5" pod "c8211333-95a0-40a6-a0cb-5bc700143cb8" (UID: "c8211333-95a0-40a6-a0cb-5bc700143cb8"). InnerVolumeSpecName "kube-api-access-bpnl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:07:46.014111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:46.014024    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpnl5\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-kube-api-access-bpnl5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:46.867654 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:46.867620    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"
Nov 03 11:07:46.902133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:46.902100    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"]
Nov 03 11:07:46.911500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:46.911463    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-d5b4t265w"]
Nov 03 11:07:47.031994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:47.031963    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8211333-95a0-40a6-a0cb-5bc700143cb8-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:07:48.233547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:48.233513    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8211333-95a0-40a6-a0cb-5bc700143cb8" path="/var/lib/kubelet/pods/c8211333-95a0-40a6-a0cb-5bc700143cb8/volumes"
Nov 03 11:07:52.837088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:52.837056    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:07:57.694213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.694170    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"]
Nov 03 11:07:57.699910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.699893    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.705086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.704689    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9f53bb-openshift-default-dockercfg-svbc9\""
Nov 03 11:07:57.709183 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.709160    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"]
Nov 03 11:07:57.878462 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.878430    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.878639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.878485    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zpb\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-kube-api-access-t7zpb\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.878691 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.878638    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-token\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.878757 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.878741    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-podinfo\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.878806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.878795    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-envoy\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.878844 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.878817    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8713c92b-ecbc-4b24-82b7-055b9dc84386-istiod-ca-cert\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.878844 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.878835    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-data\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.878988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.878975    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-certs\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.879027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.879016    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-credential-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.980539 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.980447    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zpb\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-kube-api-access-t7zpb\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.980728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.980602    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-token\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.980728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.980712    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-podinfo\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.980859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.980777    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-envoy\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.980859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.980808    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8713c92b-ecbc-4b24-82b7-055b9dc84386-istiod-ca-cert\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.980859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.980835    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-data\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.981078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.980892    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-certs\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.981078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.980991    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-credential-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.981078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.981068    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.981397 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.981363    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-data\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.981534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.981430    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-certs\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.981534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.981470    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.981534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.981515    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-credential-socket\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.982036 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.981967    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8713c92b-ecbc-4b24-82b7-055b9dc84386-istiod-ca-cert\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.983625 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.983599    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-envoy\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.983988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.983970    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-podinfo\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.989046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.989019    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zpb\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-kube-api-access-t7zpb\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:57.989210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:57.989189    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-token\") pod \"e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") " pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:58.031939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:58.024138    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:58.360868 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:07:58.360839    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8713c92b_ecbc_4b24_82b7_055b9dc84386.slice/crio-8e9dee19d54bef4c60b41f81b4af9ce2fbae44145128483d59355679e7780f9a WatchSource:0}: Error finding container 8e9dee19d54bef4c60b41f81b4af9ce2fbae44145128483d59355679e7780f9a: Status 404 returned error can't find the container with id 8e9dee19d54bef4c60b41f81b4af9ce2fbae44145128483d59355679e7780f9a
Nov 03 11:07:58.361702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:58.361673    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"]
Nov 03 11:07:58.364305 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:58.364273    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:58.364396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:58.364341    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:58.364396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:58.364371    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:07:59.002812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:59.002777    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" event={"ID":"8713c92b-ecbc-4b24-82b7-055b9dc84386","Type":"ContainerStarted","Data":"10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc"}
Nov 03 11:07:59.003368 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:59.002839    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" event={"ID":"8713c92b-ecbc-4b24-82b7-055b9dc84386","Type":"ContainerStarted","Data":"8e9dee19d54bef4c60b41f81b4af9ce2fbae44145128483d59355679e7780f9a"}
Nov 03 11:07:59.024676 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:59.024644    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:07:59.026510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:59.026484    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 11:07:59.026651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:59.026530    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 11:07:59.026700 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:07:59.026639    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podStartSLOduration=2.026625768 podStartE2EDuration="2.026625768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:07:59.022284379 +0000 UTC m=+2322.927138820" watchObservedRunningTime="2026-11-03 11:07:59.026625768 +0000 UTC m=+2322.931480209"
Nov 03 11:08:00.025306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:00.025274    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 11:08:00.025730 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:00.025320    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 11:08:01.025529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:01.025494    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 11:08:01.026007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:01.025553    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 11:08:02.029941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:02.029549    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 11:08:02.029941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:02.029626    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 11:08:03.025223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:03.025065    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 11:08:03.025223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:03.025132    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 11:08:04.025435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:04.025394    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 11:08:04.026158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:04.026122    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 11:08:05.025405 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:05.025373    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Nov 03 11:08:05.025611 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:05.025425    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Nov 03 11:08:06.029764 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:06.029725    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:08:06.030268 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:06.030253    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:08:06.031142 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:06.031124    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:08:13.190640 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:08:13.190600    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:08:19.391946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:19.379801    5393 scope.go:122] "RemoveContainer" containerID="d77632c9dacde739fc9e10e8a24743c6b38bb9c39692b8e6ec950c0772ec1fa5"
Nov 03 11:08:19.433270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:19.433242    5393 scope.go:122] "RemoveContainer" containerID="2f68401164b7a23ac52b562a62876bbd141201358a41df4b37af80c1187b2e79"
Nov 03 11:08:40.986011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:40.985907    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"]
Nov 03 11:08:40.992794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:40.992766    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.001372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.001342    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"agc-cred-a5550b35ccf4\""
Nov 03 11:08:41.001577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.001417    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"agc-cred-fc3b5b6056ca\""
Nov 03 11:08:41.002865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.002843    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"kube-root-ca.crt\""
Nov 03 11:08:41.003194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.002968    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"openshift-service-ca.crt\""
Nov 03 11:08:41.003194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.003084    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"payload-processing-external-model-e2e--90f9edc7-dockercfg-pnl4z\""
Nov 03 11:08:41.003194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.003163    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-917da2\""
Nov 03 11:08:41.003412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.003207    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"routing-overlay\""
Nov 03 11:08:41.011301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.011262    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"]
Nov 03 11:08:41.085316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.085282    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjtfz\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-kube-api-access-pjtfz\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.085527 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.085353    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.085527 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.085504    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.085674 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.085555    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.186715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.186684    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjtfz\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-kube-api-access-pjtfz\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.186949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.186745    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.186949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.186782    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.186949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.186811    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.187568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.187540    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.187685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.187601    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.189678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.189650    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.194797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.194774    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjtfz\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-kube-api-access-pjtfz\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-758545wls\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.315396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.315310    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:41.611499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.611465    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"]
Nov 03 11:08:41.612172 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:08:41.612148    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922c6a7a_0592_4b8c_9f58_ddb7c9c0fc68.slice/crio-aeb233d8328748307f20f6e64697cf2528af87ee1423677b830e53127c4de73d WatchSource:0}: Error finding container aeb233d8328748307f20f6e64697cf2528af87ee1423677b830e53127c4de73d: Status 404 returned error can't find the container with id aeb233d8328748307f20f6e64697cf2528af87ee1423677b830e53127c4de73d
Nov 03 11:08:41.645515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:41.645481    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls" event={"ID":"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68","Type":"ContainerStarted","Data":"aeb233d8328748307f20f6e64697cf2528af87ee1423677b830e53127c4de73d"}
Nov 03 11:08:42.655168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:42.655126    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls" event={"ID":"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68","Type":"ContainerStarted","Data":"08a204f2af4b10fd4997dafabafa57ed2fca2f452f2d965f704653facbc514b7"}
Nov 03 11:08:42.655676 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:42.655200    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:42.673189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:42.673132    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls" podStartSLOduration=2.673117545 podStartE2EDuration="2.673117545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:08:42.671277941 +0000 UTC m=+2366.576132382" watchObservedRunningTime="2026-11-03 11:08:42.673117545 +0000 UTC m=+2366.577971987"
Nov 03 11:08:43.192274 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:08:43.192239    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:08:51.850825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:51.850792    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"]
Nov 03 11:08:52.632525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:52.632491    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"]
Nov 03 11:08:53.104905 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.104843    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"]
Nov 03 11:08:53.118196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.118165    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"]
Nov 03 11:08:53.120596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.120567    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.127678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.127635    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"agc-cred-a9d681f7500c\""
Nov 03 11:08:53.127678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.127661    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-35936b\"/\"payload-processing-external-model-e2e--ac720701-dockercfg-w6t4p\""
Nov 03 11:08:53.253516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.253458    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.269769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.260420    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zctk\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-kube-api-access-2zctk\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.279272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.279067    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.279637 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.279612    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.390994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.388408    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.390994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.388573    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zctk\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-kube-api-access-2zctk\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.390994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.388711    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.390994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.388734    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.391693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.391658    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.392523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.392502    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.411953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.408736    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.449958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.447381    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zctk\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-kube-api-access-2zctk\") pod \"payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") " pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:53.717677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.717590    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:08:53.738301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:53.738268    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:54.051652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:54.051621    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"]
Nov 03 11:08:54.052273 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:08:54.052238    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba085a8_852b_469d_a0df_b2c2dd3502b2.slice/crio-e12d3a01dd066dc8113f43d3157bff70ce6cad985e2ab0dd7fe1d71c6722c291 WatchSource:0}: Error finding container e12d3a01dd066dc8113f43d3157bff70ce6cad985e2ab0dd7fe1d71c6722c291: Status 404 returned error can't find the container with id e12d3a01dd066dc8113f43d3157bff70ce6cad985e2ab0dd7fe1d71c6722c291
Nov 03 11:08:54.891373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:54.891335    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6" event={"ID":"4ba085a8-852b-469d-a0df-b2c2dd3502b2","Type":"ContainerStarted","Data":"fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799"}
Nov 03 11:08:54.891807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:54.891394    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:08:54.891807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:54.891419    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6" event={"ID":"4ba085a8-852b-469d-a0df-b2c2dd3502b2","Type":"ContainerStarted","Data":"e12d3a01dd066dc8113f43d3157bff70ce6cad985e2ab0dd7fe1d71c6722c291"}
Nov 03 11:08:54.911549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:54.911490    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6" podStartSLOduration=1.911474094 podStartE2EDuration="1.911474094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:08:54.906806398 +0000 UTC m=+2378.811660842" watchObservedRunningTime="2026-11-03 11:08:54.911474094 +0000 UTC m=+2378.816328534"
Nov 03 11:08:56.852231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:56.852149    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx" podUID="72cf8b4c-abb1-470d-a4ca-29abe60e99dd" containerName="payload-processing" containerID="cri-o://8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e" gracePeriod=25
Nov 03 11:08:57.138553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.138526    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:08:57.166272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.166233    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-plugins-config-volume\") pod \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") "
Nov 03 11:08:57.166517 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.166495    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-kube-api-access-q5v4k\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-kube-api-access-q5v4k\") pod \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") "
Nov 03 11:08:57.166660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.166643    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-routing-overlay-volume\") pod \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") "
Nov 03 11:08:57.166776 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.166758    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-provider-credentials\") pod \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\" (UID: \"72cf8b4c-abb1-470d-a4ca-29abe60e99dd\") "
Nov 03 11:08:57.169317 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.168524    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-plugins-config-volume" pod "72cf8b4c-abb1-470d-a4ca-29abe60e99dd" (UID: "72cf8b4c-abb1-470d-a4ca-29abe60e99dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:08:57.169317 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.168949    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-routing-overlay-volume" pod "72cf8b4c-abb1-470d-a4ca-29abe60e99dd" (UID: "72cf8b4c-abb1-470d-a4ca-29abe60e99dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:08:57.171532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.171507    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-provider-credentials" pod "72cf8b4c-abb1-470d-a4ca-29abe60e99dd" (UID: "72cf8b4c-abb1-470d-a4ca-29abe60e99dd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:08:57.172984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.172902    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-kube-api-access-q5v4k" pod "72cf8b4c-abb1-470d-a4ca-29abe60e99dd" (UID: "72cf8b4c-abb1-470d-a4ca-29abe60e99dd"). InnerVolumeSpecName "kube-api-access-q5v4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:08:57.269648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.269605    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5v4k\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-kube-api-access-q5v4k\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:57.269648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.269638    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:57.269648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.269654    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:57.269937 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.269667    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72cf8b4c-abb1-470d-a4ca-29abe60e99dd-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:57.634112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.634071    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs" podUID="c2e9eb33-2fb8-4241-85aa-3c600c1d197d" containerName="payload-processing" containerID="cri-o://8141c2173a0003312a41e9030930601cb4391afbda4891f423947a12df061bf6" gracePeriod=25
Nov 03 11:08:57.917306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.917279    5393 generic.go:356] "Generic (PLEG): container finished" podID="c2e9eb33-2fb8-4241-85aa-3c600c1d197d" containerID="8141c2173a0003312a41e9030930601cb4391afbda4891f423947a12df061bf6" exitCode=0
Nov 03 11:08:57.917739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.917359    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs" event={"ID":"c2e9eb33-2fb8-4241-85aa-3c600c1d197d","Type":"ContainerDied","Data":"8141c2173a0003312a41e9030930601cb4391afbda4891f423947a12df061bf6"}
Nov 03 11:08:57.917739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.917466    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs" event={"ID":"c2e9eb33-2fb8-4241-85aa-3c600c1d197d","Type":"ContainerDied","Data":"a724c4b8faaffd65c9bd788a76bf3dbdfdb3d7d423fb643e366439605b9b717e"}
Nov 03 11:08:57.917739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.917492    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a724c4b8faaffd65c9bd788a76bf3dbdfdb3d7d423fb643e366439605b9b717e"
Nov 03 11:08:57.919144 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.919120    5393 generic.go:356] "Generic (PLEG): container finished" podID="72cf8b4c-abb1-470d-a4ca-29abe60e99dd" containerID="8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e" exitCode=0
Nov 03 11:08:57.919247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.919195    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"
Nov 03 11:08:57.919302 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.919196    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx" event={"ID":"72cf8b4c-abb1-470d-a4ca-29abe60e99dd","Type":"ContainerDied","Data":"8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e"}
Nov 03 11:08:57.919342 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.919312    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx" event={"ID":"72cf8b4c-abb1-470d-a4ca-29abe60e99dd","Type":"ContainerDied","Data":"5ef0e99664d2f013ae60eae19c13e1f95621c49c70f8a7f2a0326d4d990f4d25"}
Nov 03 11:08:57.919342 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.919335    5393 scope.go:122] "RemoveContainer" containerID="8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e"
Nov 03 11:08:57.941416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.941400    5393 scope.go:122] "RemoveContainer" containerID="8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e"
Nov 03 11:08:57.941957 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:08:57.941905    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e\": container with ID starting with 8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e not found: ID does not exist" containerID="8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e"
Nov 03 11:08:57.942067 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.941953    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e"} err="failed to get container status \"8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e\": rpc error: code = NotFound desc = could not find container \"8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e\": container with ID starting with 8723de449dd13886dbb4f7179bf3d9818b50a27c0e981a3f5bbcf0758cae469e not found: ID does not exist"
Nov 03 11:08:57.946177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.946158    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:08:57.960962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.960902    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"]
Nov 03 11:08:57.964682 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.964659    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-694dspdkx"]
Nov 03 11:08:57.982631 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.982599    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-kube-api-access-ffmvf\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-kube-api-access-ffmvf\") pod \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") "
Nov 03 11:08:57.982793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.982698    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-routing-overlay-volume\") pod \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") "
Nov 03 11:08:57.982868 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.982841    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-plugins-config-volume\") pod \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") "
Nov 03 11:08:57.983011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.982987    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-provider-credentials\") pod \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\" (UID: \"c2e9eb33-2fb8-4241-85aa-3c600c1d197d\") "
Nov 03 11:08:57.983581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.983302    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-routing-overlay-volume" pod "c2e9eb33-2fb8-4241-85aa-3c600c1d197d" (UID: "c2e9eb33-2fb8-4241-85aa-3c600c1d197d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:08:57.983882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.983852    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:57.988967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.984430    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-plugins-config-volume" pod "c2e9eb33-2fb8-4241-85aa-3c600c1d197d" (UID: "c2e9eb33-2fb8-4241-85aa-3c600c1d197d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:08:57.988967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.986889    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-provider-credentials" pod "c2e9eb33-2fb8-4241-85aa-3c600c1d197d" (UID: "c2e9eb33-2fb8-4241-85aa-3c600c1d197d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:08:57.989356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:57.989332    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-kube-api-access-ffmvf" pod "c2e9eb33-2fb8-4241-85aa-3c600c1d197d" (UID: "c2e9eb33-2fb8-4241-85aa-3c600c1d197d"). InnerVolumeSpecName "kube-api-access-ffmvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:08:58.085336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:58.085301    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffmvf\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-kube-api-access-ffmvf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:58.085336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:58.085338    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:58.085539 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:58.085353    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2e9eb33-2fb8-4241-85aa-3c600c1d197d-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:08:58.232048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:58.231967    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cf8b4c-abb1-470d-a4ca-29abe60e99dd" path="/var/lib/kubelet/pods/72cf8b4c-abb1-470d-a4ca-29abe60e99dd/volumes"
Nov 03 11:08:58.928594 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:58.928556    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"
Nov 03 11:08:58.948416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:58.948386    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"]
Nov 03 11:08:58.953463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:08:58.953438    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-594bwd8qs"]
Nov 03 11:09:00.233215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:00.233178    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e9eb33-2fb8-4241-85aa-3c600c1d197d" path="/var/lib/kubelet/pods/c2e9eb33-2fb8-4241-85aa-3c600c1d197d/volumes"
Nov 03 11:09:03.434686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:03.434353    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"]
Nov 03 11:09:03.439145 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:03.439119    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:09:08.441977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.435426    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6" podUID="4ba085a8-852b-469d-a0df-b2c2dd3502b2" containerName="payload-processing" containerID="cri-o://fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799" gracePeriod=25
Nov 03 11:09:08.725863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.725833    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:09:08.886478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.875260    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-routing-overlay-volume\") pod \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") "
Nov 03 11:09:08.886478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.875672    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-routing-overlay-volume" pod "4ba085a8-852b-469d-a0df-b2c2dd3502b2" (UID: "4ba085a8-852b-469d-a0df-b2c2dd3502b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:08.886478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.875997    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-plugins-config-volume\") pod \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") "
Nov 03 11:09:08.886478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.876414    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-plugins-config-volume" pod "4ba085a8-852b-469d-a0df-b2c2dd3502b2" (UID: "4ba085a8-852b-469d-a0df-b2c2dd3502b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:08.886478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.876579    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-provider-credentials\") pod \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") "
Nov 03 11:09:08.905967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.899591    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-kube-api-access-2zctk\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-kube-api-access-2zctk\") pod \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\" (UID: \"4ba085a8-852b-469d-a0df-b2c2dd3502b2\") "
Nov 03 11:09:08.905967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.900576    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:08.905967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:08.900603    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba085a8-852b-469d-a0df-b2c2dd3502b2-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:09.028280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.028170    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-kube-api-access-2zctk" pod "4ba085a8-852b-469d-a0df-b2c2dd3502b2" (UID: "4ba085a8-852b-469d-a0df-b2c2dd3502b2"). InnerVolumeSpecName "kube-api-access-2zctk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:09.165879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.132120    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-provider-credentials" pod "4ba085a8-852b-469d-a0df-b2c2dd3502b2" (UID: "4ba085a8-852b-469d-a0df-b2c2dd3502b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:09.196944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.168875    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zctk\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-kube-api-access-2zctk\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:09.198328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.198294    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba085a8-852b-469d-a0df-b2c2dd3502b2-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:09.295342 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.295221    5393 generic.go:356] "Generic (PLEG): container finished" podID="4ba085a8-852b-469d-a0df-b2c2dd3502b2" containerID="fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799" exitCode=0
Nov 03 11:09:09.297635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.297604    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"
Nov 03 11:09:09.329590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.297826    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6" event={"ID":"4ba085a8-852b-469d-a0df-b2c2dd3502b2","Type":"ContainerDied","Data":"fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799"}
Nov 03 11:09:09.340580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.340540    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6" event={"ID":"4ba085a8-852b-469d-a0df-b2c2dd3502b2","Type":"ContainerDied","Data":"e12d3a01dd066dc8113f43d3157bff70ce6cad985e2ab0dd7fe1d71c6722c291"}
Nov 03 11:09:09.363595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.363563    5393 scope.go:122] "RemoveContainer" containerID="fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799"
Nov 03 11:09:09.390948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.387966    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"]
Nov 03 11:09:09.390948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.388030    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-35936b/payload-processing-external-model-e2e-extmod-a-35936b-688fsrnt6"]
Nov 03 11:09:09.427940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.424681    5393 scope.go:122] "RemoveContainer" containerID="fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799"
Nov 03 11:09:09.430564 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:09:09.430534    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799\": container with ID starting with fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799 not found: ID does not exist" containerID="fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799"
Nov 03 11:09:09.430804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:09.430776    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799"} err="failed to get container status \"fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799\": rpc error: code = NotFound desc = could not find container \"fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799\": container with ID starting with fdf3a58fba0b04a1a1c5141a840bbe81055cef293d9518092f86ff45a84e9799 not found: ID does not exist"
Nov 03 11:09:10.257279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:10.257246    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba085a8-852b-469d-a0df-b2c2dd3502b2" path="/var/lib/kubelet/pods/4ba085a8-852b-469d-a0df-b2c2dd3502b2/volumes"
Nov 03 11:09:13.207042 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:09:13.207006    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:09:17.763939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.763746    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:09:17.790806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.790772    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.960027    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"]
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961268    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4ba085a8-852b-469d-a0df-b2c2dd3502b2" containerName="payload-processing"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961289    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba085a8-852b-469d-a0df-b2c2dd3502b2" containerName="payload-processing"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961330    5393 cpu_manager.go:414] "containerMap: removing container" podUID="72cf8b4c-abb1-470d-a4ca-29abe60e99dd" containerName="payload-processing"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961339    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cf8b4c-abb1-470d-a4ca-29abe60e99dd" containerName="payload-processing"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961406    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c2e9eb33-2fb8-4241-85aa-3c600c1d197d" containerName="payload-processing"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961416    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e9eb33-2fb8-4241-85aa-3c600c1d197d" containerName="payload-processing"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961585    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e9eb33-2fb8-4241-85aa-3c600c1d197d" containerName="payload-processing"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961603    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba085a8-852b-469d-a0df-b2c2dd3502b2" containerName="payload-processing"
Nov 03 11:09:17.964282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.961621    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cf8b4c-abb1-470d-a4ca-29abe60e99dd" containerName="payload-processing"
Nov 03 11:09:17.966610 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.966572    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"]
Nov 03 11:09:17.966854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.966699    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:17.971570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.971522    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ec531588fd48\""
Nov 03 11:09:17.972560 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.972537    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 11:09:17.972688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.972623    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 11:09:17.972769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.972749    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pjcw6\""
Nov 03 11:09:17.972909 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.972851    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 11:09:17.973015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:17.972979    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 11:09:18.165943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.162655    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-routing-overlay-volume\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.165943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.162776    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-plugins-config-volume\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.165943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.162941    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-provider-credentials\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.165943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.163225    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm97s\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-kube-api-access-bm97s\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.271943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.271271    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm97s\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-kube-api-access-bm97s\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.271943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.271465    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-routing-overlay-volume\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.284976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.278698    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-routing-overlay-volume\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.284976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.278808    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-plugins-config-volume\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.284976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.278997    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-provider-credentials\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.297941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.288024    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-plugins-config-volume\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.323948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.315114    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm97s\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-kube-api-access-bm97s\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.323948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.316757    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-provider-credentials\") pod \"payload-processing-external-model-6b76f7948b-5t8cf\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") " pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.596690 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.596657    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:18.893852 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:09:18.893823    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef711413_5409_4833_a93f_27a2cd8849e3.slice/crio-5f60c324a09bc4eb37086e12fa5182ef42df88e4747bf9db0a893945b67531f4 WatchSource:0}: Error finding container 5f60c324a09bc4eb37086e12fa5182ef42df88e4747bf9db0a893945b67531f4: Status 404 returned error can't find the container with id 5f60c324a09bc4eb37086e12fa5182ef42df88e4747bf9db0a893945b67531f4
Nov 03 11:09:18.894328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:18.894097    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"]
Nov 03 11:09:19.466329 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:19.466293    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf" event={"ID":"ef711413-5409-4833-a93f-27a2cd8849e3","Type":"ContainerStarted","Data":"e851a9e473c1a0235b7b0b541816e5aa63871efe22f251b8a34c02ef255b9c49"}
Nov 03 11:09:19.466488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:19.466348    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf" event={"ID":"ef711413-5409-4833-a93f-27a2cd8849e3","Type":"ContainerStarted","Data":"5f60c324a09bc4eb37086e12fa5182ef42df88e4747bf9db0a893945b67531f4"}
Nov 03 11:09:19.466549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:19.466513    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:19.483320 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:19.483268    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf" podStartSLOduration=2.483256258 podStartE2EDuration="2.483256258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:19.481858194 +0000 UTC m=+2403.386712637" watchObservedRunningTime="2026-11-03 11:09:19.483256258 +0000 UTC m=+2403.388110699"
Nov 03 11:09:30.478800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:30.478768    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:09:33.234946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.231030    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"]
Nov 03 11:09:33.258942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.258340    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.267948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.260064    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"]
Nov 03 11:09:33.283943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.282812    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"agc-cred-8d95d0be60f9\""
Nov 03 11:09:33.283943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.282962    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9f53bb\""
Nov 03 11:09:33.293041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.285295    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"openshift-service-ca.crt\""
Nov 03 11:09:33.293041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.285425    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"routing-overlay\""
Nov 03 11:09:33.293041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.285643    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"kube-root-ca.crt\""
Nov 03 11:09:33.329944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.313052    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-e2e--b43e39c2-dockercfg-6rngp\""
Nov 03 11:09:33.416367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.416331    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.416785 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.416749    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.417074 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.417052    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r45v\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-kube-api-access-5r45v\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.417158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.417094    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.518621 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.518527    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.518796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.518650    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r45v\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-kube-api-access-5r45v\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.518796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.518675    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.518796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.518763    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.519590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.519559    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.519731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.519647    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.521289 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.521267    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.526463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.526441    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r45v\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-kube-api-access-5r45v\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.661530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.661492    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:33.949050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:33.949018    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"]
Nov 03 11:09:33.949175 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:09:33.949156    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99027886_1116_489c_bb93_70a52a0423a7.slice/crio-4381a28fd24a2877b1c76b155cbff6a1e490a63146e224e098f835fd5019956a WatchSource:0}: Error finding container 4381a28fd24a2877b1c76b155cbff6a1e490a63146e224e098f835fd5019956a: Status 404 returned error can't find the container with id 4381a28fd24a2877b1c76b155cbff6a1e490a63146e224e098f835fd5019956a
Nov 03 11:09:34.603696 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:34.603664    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk" event={"ID":"99027886-1116-489c-bb93-70a52a0423a7","Type":"ContainerStarted","Data":"87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188"}
Nov 03 11:09:34.604222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:34.603717    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk" event={"ID":"99027886-1116-489c-bb93-70a52a0423a7","Type":"ContainerStarted","Data":"4381a28fd24a2877b1c76b155cbff6a1e490a63146e224e098f835fd5019956a"}
Nov 03 11:09:34.604222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:34.603739    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:34.622534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:34.622482    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk" podStartSLOduration=1.6224685060000001 podStartE2EDuration="1.622468506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:34.619789993 +0000 UTC m=+2418.524644434" watchObservedRunningTime="2026-11-03 11:09:34.622468506 +0000 UTC m=+2418.527322947"
Nov 03 11:09:36.575101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.575064    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"]
Nov 03 11:09:36.579725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.579700    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.587570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.587539    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"]
Nov 03 11:09:36.638943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.632327    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.638943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.632395    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2m2\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-kube-api-access-mz2m2\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.638943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.632529    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.769132 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.769099    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.875941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.871284    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.875941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.871466    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.875941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.871506    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2m2\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-kube-api-access-mz2m2\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.875941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.871608    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.875941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.872890    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.875941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.873695    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.901195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.894969    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2m2\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-kube-api-access-mz2m2\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:36.924940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:36.918981    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:37.198578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:37.198488    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:37.488793 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:09:37.488760    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9091470e_69f9_450d_978d_9ae10f0a1bbe.slice/crio-f207560c19b20e74d8d9b9c33ad306c604a586e771b4af6fded7d74c295a5e12 WatchSource:0}: Error finding container f207560c19b20e74d8d9b9c33ad306c604a586e771b4af6fded7d74c295a5e12: Status 404 returned error can't find the container with id f207560c19b20e74d8d9b9c33ad306c604a586e771b4af6fded7d74c295a5e12
Nov 03 11:09:37.489403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:37.489371    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"]
Nov 03 11:09:37.816036 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:37.815887    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77" event={"ID":"9091470e-69f9-450d-978d-9ae10f0a1bbe","Type":"ContainerStarted","Data":"70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009"}
Nov 03 11:09:37.816036 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:37.816004    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77" event={"ID":"9091470e-69f9-450d-978d-9ae10f0a1bbe","Type":"ContainerStarted","Data":"f207560c19b20e74d8d9b9c33ad306c604a586e771b4af6fded7d74c295a5e12"}
Nov 03 11:09:37.816583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:37.816361    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:37.834098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:37.834041    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77" podStartSLOduration=1.834026713 podStartE2EDuration="1.834026713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:09:37.832533887 +0000 UTC m=+2421.737388330" watchObservedRunningTime="2026-11-03 11:09:37.834026713 +0000 UTC m=+2421.738881170"
Nov 03 11:09:43.196696 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:09:43.196656    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:09:44.176564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:44.176525    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"]
Nov 03 11:09:44.186525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:44.186491    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:48.766627 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:48.766591    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"]
Nov 03 11:09:48.829511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:48.829480    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:09:48.880455 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:48.880419    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"]
Nov 03 11:09:49.181947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.180983    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk" podUID="99027886-1116-489c-bb93-70a52a0423a7" containerName="payload-processing" containerID="cri-o://87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188" gracePeriod=25
Nov 03 11:09:49.696173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.696096    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:49.774021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.773983    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-routing-overlay-volume\") pod \"99027886-1116-489c-bb93-70a52a0423a7\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") "
Nov 03 11:09:49.782869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.782836    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-routing-overlay-volume" pod "99027886-1116-489c-bb93-70a52a0423a7" (UID: "99027886-1116-489c-bb93-70a52a0423a7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:49.783054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.782947    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-kube-api-access-5r45v\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-kube-api-access-5r45v\") pod \"99027886-1116-489c-bb93-70a52a0423a7\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") "
Nov 03 11:09:49.783054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.783004    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-plugins-config-volume\") pod \"99027886-1116-489c-bb93-70a52a0423a7\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") "
Nov 03 11:09:49.783219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.783077    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-provider-credentials\") pod \"99027886-1116-489c-bb93-70a52a0423a7\" (UID: \"99027886-1116-489c-bb93-70a52a0423a7\") "
Nov 03 11:09:49.783790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.783763    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-plugins-config-volume" pod "99027886-1116-489c-bb93-70a52a0423a7" (UID: "99027886-1116-489c-bb93-70a52a0423a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:49.784089 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.784071    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:49.784189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.784094    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99027886-1116-489c-bb93-70a52a0423a7-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:49.786506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.786391    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-kube-api-access-5r45v" pod "99027886-1116-489c-bb93-70a52a0423a7" (UID: "99027886-1116-489c-bb93-70a52a0423a7"). InnerVolumeSpecName "kube-api-access-5r45v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:49.786611 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.786539    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-provider-credentials" pod "99027886-1116-489c-bb93-70a52a0423a7" (UID: "99027886-1116-489c-bb93-70a52a0423a7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:49.885652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.885612    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r45v\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-kube-api-access-5r45v\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:49.885652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:49.885644    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99027886-1116-489c-bb93-70a52a0423a7-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:50.164743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.164706    5393 generic.go:356] "Generic (PLEG): container finished" podID="99027886-1116-489c-bb93-70a52a0423a7" containerID="87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188" exitCode=0
Nov 03 11:09:50.164946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.164757    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"
Nov 03 11:09:50.164946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.164793    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk" event={"ID":"99027886-1116-489c-bb93-70a52a0423a7","Type":"ContainerDied","Data":"87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188"}
Nov 03 11:09:50.164946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.164856    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk" event={"ID":"99027886-1116-489c-bb93-70a52a0423a7","Type":"ContainerDied","Data":"4381a28fd24a2877b1c76b155cbff6a1e490a63146e224e098f835fd5019956a"}
Nov 03 11:09:50.164946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.164887    5393 scope.go:122] "RemoveContainer" containerID="87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188"
Nov 03 11:09:50.177488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.177460    5393 scope.go:122] "RemoveContainer" containerID="87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188"
Nov 03 11:09:50.177861 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:09:50.177838    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188\": container with ID starting with 87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188 not found: ID does not exist" containerID="87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188"
Nov 03 11:09:50.177997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.177869    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188"} err="failed to get container status \"87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188\": rpc error: code = NotFound desc = could not find container \"87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188\": container with ID starting with 87974e078aa76f70d6a500605fef13f59e949c3848aa5064447da58dbabec188 not found: ID does not exist"
Nov 03 11:09:50.190435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.190401    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"]
Nov 03 11:09:50.194638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.194609    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-747f65vbk"]
Nov 03 11:09:50.234568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:50.234533    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99027886-1116-489c-bb93-70a52a0423a7" path="/var/lib/kubelet/pods/99027886-1116-489c-bb93-70a52a0423a7/volumes"
Nov 03 11:09:53.767244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:53.767189    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c" podUID="f49c2cff-90a0-4fb7-bf77-c63cc239e69f" containerName="payload-processing" containerID="cri-o://f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92" gracePeriod=25
Nov 03 11:09:53.883377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:53.883325    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls" podUID="922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" containerName="payload-processing" containerID="cri-o://08a204f2af4b10fd4997dafabafa57ed2fca2f452f2d965f704653facbc514b7" gracePeriod=25
Nov 03 11:09:54.144079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.144049    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:09:54.202828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.202798    5393 generic.go:356] "Generic (PLEG): container finished" podID="922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" containerID="08a204f2af4b10fd4997dafabafa57ed2fca2f452f2d965f704653facbc514b7" exitCode=0
Nov 03 11:09:54.203069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.202986    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls" event={"ID":"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68","Type":"ContainerDied","Data":"08a204f2af4b10fd4997dafabafa57ed2fca2f452f2d965f704653facbc514b7"}
Nov 03 11:09:54.204734 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.204706    5393 generic.go:356] "Generic (PLEG): container finished" podID="f49c2cff-90a0-4fb7-bf77-c63cc239e69f" containerID="f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92" exitCode=0
Nov 03 11:09:54.204860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.204822    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"
Nov 03 11:09:54.204934 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.204862    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c" event={"ID":"f49c2cff-90a0-4fb7-bf77-c63cc239e69f","Type":"ContainerDied","Data":"f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92"}
Nov 03 11:09:54.204991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.204963    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c" event={"ID":"f49c2cff-90a0-4fb7-bf77-c63cc239e69f","Type":"ContainerDied","Data":"228462269fe247341b9f571e253f3943cc775cb6a82e26f216de51cb67312d46"}
Nov 03 11:09:54.205043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.205002    5393 scope.go:122] "RemoveContainer" containerID="f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92"
Nov 03 11:09:54.220793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.220769    5393 scope.go:122] "RemoveContainer" containerID="f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92"
Nov 03 11:09:54.221206 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:09:54.221176    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92\": container with ID starting with f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92 not found: ID does not exist" containerID="f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92"
Nov 03 11:09:54.221321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.221222    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92"} err="failed to get container status \"f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92\": rpc error: code = NotFound desc = could not find container \"f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92\": container with ID starting with f501354b224d0abfb3b172357278c8061c4b5943d72f934280975a92120b5d92 not found: ID does not exist"
Nov 03 11:09:54.270461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.270436    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:09:54.310229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.310147    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-kube-api-access-86c7c\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-kube-api-access-86c7c\") pod \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") "
Nov 03 11:09:54.310450 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.310430    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-provider-credentials\") pod \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") "
Nov 03 11:09:54.310511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.310479    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-plugins-config-volume\") pod \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") "
Nov 03 11:09:54.310679 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.310653    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-routing-overlay-volume\") pod \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\" (UID: \"f49c2cff-90a0-4fb7-bf77-c63cc239e69f\") "
Nov 03 11:09:54.311792 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.311243    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-plugins-config-volume" pod "f49c2cff-90a0-4fb7-bf77-c63cc239e69f" (UID: "f49c2cff-90a0-4fb7-bf77-c63cc239e69f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:54.311792 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.311264    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-routing-overlay-volume" pod "f49c2cff-90a0-4fb7-bf77-c63cc239e69f" (UID: "f49c2cff-90a0-4fb7-bf77-c63cc239e69f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:54.312310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.312281    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:54.312310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.312307    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:54.313506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.313482    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-provider-credentials" pod "f49c2cff-90a0-4fb7-bf77-c63cc239e69f" (UID: "f49c2cff-90a0-4fb7-bf77-c63cc239e69f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:54.313627 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.313524    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-kube-api-access-86c7c" pod "f49c2cff-90a0-4fb7-bf77-c63cc239e69f" (UID: "f49c2cff-90a0-4fb7-bf77-c63cc239e69f"). InnerVolumeSpecName "kube-api-access-86c7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:54.413354 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.413314    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-kube-api-access-pjtfz\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-kube-api-access-pjtfz\") pod \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") "
Nov 03 11:09:54.413571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.413377    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-plugins-config-volume\") pod \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") "
Nov 03 11:09:54.413571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.413407    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-provider-credentials\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-provider-credentials\") pod \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") "
Nov 03 11:09:54.413571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.413504    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-routing-overlay-volume\") pod \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\" (UID: \"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68\") "
Nov 03 11:09:54.413887 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.413854    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-plugins-config-volume" pod "922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" (UID: "922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:54.414484 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.414458    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-routing-overlay-volume" pod "922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" (UID: "922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:09:54.414587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.414491    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:54.414587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.414521    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:54.414587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.414537    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86c7c\" (UniqueName: \"kubernetes.io/projected/f49c2cff-90a0-4fb7-bf77-c63cc239e69f-kube-api-access-86c7c\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:54.416250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.416230    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-kube-api-access-pjtfz" pod "922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" (UID: "922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68"). InnerVolumeSpecName "kube-api-access-pjtfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:54.417828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.417800    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-provider-credentials" pod "922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" (UID: "922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:09:54.515802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.515773    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjtfz\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-kube-api-access-pjtfz\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:54.515802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.515804    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:54.515989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.515815    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:09:54.531143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.531113    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"]
Nov 03 11:09:54.534784 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:54.534751    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-185908/payload-processing-external-model-e2e-extmod-a-185908-6955szd5c"]
Nov 03 11:09:55.214121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:55.214090    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"
Nov 03 11:09:55.214121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:55.214113    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls" event={"ID":"922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68","Type":"ContainerDied","Data":"aeb233d8328748307f20f6e64697cf2528af87ee1423677b830e53127c4de73d"}
Nov 03 11:09:55.214693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:55.214217    5393 scope.go:122] "RemoveContainer" containerID="08a204f2af4b10fd4997dafabafa57ed2fca2f452f2d965f704653facbc514b7"
Nov 03 11:09:55.241667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:55.241631    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"]
Nov 03 11:09:55.249265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:55.249240    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-758545wls"]
Nov 03 11:09:56.232037 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:56.232002    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" path="/var/lib/kubelet/pods/922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68/volumes"
Nov 03 11:09:56.232536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:56.232521    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f49c2cff-90a0-4fb7-bf77-c63cc239e69f" path="/var/lib/kubelet/pods/f49c2cff-90a0-4fb7-bf77-c63cc239e69f/volumes"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.747490    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"]
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.748759    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f49c2cff-90a0-4fb7-bf77-c63cc239e69f" containerName="payload-processing"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.748780    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49c2cff-90a0-4fb7-bf77-c63cc239e69f" containerName="payload-processing"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.748825    5393 cpu_manager.go:414] "containerMap: removing container" podUID="99027886-1116-489c-bb93-70a52a0423a7" containerName="payload-processing"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.748833    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99027886-1116-489c-bb93-70a52a0423a7" containerName="payload-processing"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.748861    5393 cpu_manager.go:414] "containerMap: removing container" podUID="922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" containerName="payload-processing"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.748869    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" containerName="payload-processing"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.749106    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49c2cff-90a0-4fb7-bf77-c63cc239e69f" containerName="payload-processing"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.749129    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="99027886-1116-489c-bb93-70a52a0423a7" containerName="payload-processing"
Nov 03 11:09:58.750948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.749141    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="922c6a7a-0592-4b8c-9f58-ddb7c9c0fc68" containerName="payload-processing"
Nov 03 11:09:58.760949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.759263    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.771984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.767997    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fd1ec75a88bc\""
Nov 03 11:09:58.771984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.768142    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-abf28fc53a90\""
Nov 03 11:09:58.791001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.777651    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"]
Nov 03 11:09:58.863429 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.863377    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcnn\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-kube-api-access-jdcnn\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.863641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.863492    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-routing-overlay-volume\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.863641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.863531    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-provider-credentials\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.863800 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.863781    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-plugins-config-volume\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.965150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.965115    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-plugins-config-volume\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.965355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.965182    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcnn\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-kube-api-access-jdcnn\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.965355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.965234    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-routing-overlay-volume\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.965355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.965255    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-provider-credentials\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.965895 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.965869    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-routing-overlay-volume\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.966025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.965959    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-plugins-config-volume\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.968047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.968024    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-provider-credentials\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:58.973204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:58.973183    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcnn\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-kube-api-access-jdcnn\") pod \"payload-processing-external-model-8444774d4b-dvzrf\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") " pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:59.152787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:59.152750    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:09:59.453447 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:59.453419    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"]
Nov 03 11:09:59.454077 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:09:59.454052    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5 WatchSource:0}: Error finding container b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5: Status 404 returned error can't find the container with id b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5
Nov 03 11:09:59.827036 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:09:59.826941    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"]
Nov 03 11:10:00.277994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:00.277952    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf" event={"ID":"536d109b-d30e-49e4-aed0-59eb676c5830","Type":"ContainerStarted","Data":"5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014"}
Nov 03 11:10:00.278283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:00.278264    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf" event={"ID":"536d109b-d30e-49e4-aed0-59eb676c5830","Type":"ContainerStarted","Data":"b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5"}
Nov 03 11:10:00.279340 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:00.279299    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:10:00.306005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:00.298511    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf" podStartSLOduration=2.298489255 podStartE2EDuration="2.298489255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:00.295588701 +0000 UTC m=+2444.200443142" watchObservedRunningTime="2026-11-03 11:10:00.298489255 +0000 UTC m=+2444.203343699"
Nov 03 11:10:04.828015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:04.827951    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77" podUID="9091470e-69f9-450d-978d-9ae10f0a1bbe" containerName="payload-processing" containerID="cri-o://70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009" gracePeriod=25
Nov 03 11:10:05.188393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.188360    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:10:05.321024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.320984    5393 generic.go:356] "Generic (PLEG): container finished" podID="9091470e-69f9-450d-978d-9ae10f0a1bbe" containerID="70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009" exitCode=0
Nov 03 11:10:05.321215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.321051    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"
Nov 03 11:10:05.321215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.321074    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77" event={"ID":"9091470e-69f9-450d-978d-9ae10f0a1bbe","Type":"ContainerDied","Data":"70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009"}
Nov 03 11:10:05.321215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.321135    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77" event={"ID":"9091470e-69f9-450d-978d-9ae10f0a1bbe","Type":"ContainerDied","Data":"f207560c19b20e74d8d9b9c33ad306c604a586e771b4af6fded7d74c295a5e12"}
Nov 03 11:10:05.321215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.321158    5393 scope.go:122] "RemoveContainer" containerID="70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009"
Nov 03 11:10:05.337503 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.337483    5393 scope.go:122] "RemoveContainer" containerID="70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009"
Nov 03 11:10:05.338023 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:05.337996    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009\": container with ID starting with 70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009 not found: ID does not exist" containerID="70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009"
Nov 03 11:10:05.338106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.338030    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009"} err="failed to get container status \"70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009\": rpc error: code = NotFound desc = could not find container \"70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009\": container with ID starting with 70ae0eb554a5982e9f3fee4203afa1e26ec75fb1537c25411f58051e08a08009 not found: ID does not exist"
Nov 03 11:10:05.339468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.339443    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-routing-overlay-volume\") pod \"9091470e-69f9-450d-978d-9ae10f0a1bbe\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") "
Nov 03 11:10:05.339604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.339589    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-provider-credentials\") pod \"9091470e-69f9-450d-978d-9ae10f0a1bbe\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") "
Nov 03 11:10:05.339821 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.339804    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-kube-api-access-mz2m2\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-kube-api-access-mz2m2\") pod \"9091470e-69f9-450d-978d-9ae10f0a1bbe\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") "
Nov 03 11:10:05.339961 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.339940    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-routing-overlay-volume" pod "9091470e-69f9-450d-978d-9ae10f0a1bbe" (UID: "9091470e-69f9-450d-978d-9ae10f0a1bbe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:05.340016 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.340003    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-plugins-config-volume\") pod \"9091470e-69f9-450d-978d-9ae10f0a1bbe\" (UID: \"9091470e-69f9-450d-978d-9ae10f0a1bbe\") "
Nov 03 11:10:05.340516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.340439    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-plugins-config-volume" pod "9091470e-69f9-450d-978d-9ae10f0a1bbe" (UID: "9091470e-69f9-450d-978d-9ae10f0a1bbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:05.340716 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.340689    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:05.340716 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.340702    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9091470e-69f9-450d-978d-9ae10f0a1bbe-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:05.342261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.342239    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-kube-api-access-mz2m2" pod "9091470e-69f9-450d-978d-9ae10f0a1bbe" (UID: "9091470e-69f9-450d-978d-9ae10f0a1bbe"). InnerVolumeSpecName "kube-api-access-mz2m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:05.342345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.342269    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-provider-credentials" pod "9091470e-69f9-450d-978d-9ae10f0a1bbe" (UID: "9091470e-69f9-450d-978d-9ae10f0a1bbe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:05.442822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.442738    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:05.442822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.442766    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz2m2\" (UniqueName: \"kubernetes.io/projected/9091470e-69f9-450d-978d-9ae10f0a1bbe-kube-api-access-mz2m2\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:05.649893 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.649854    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"]
Nov 03 11:10:05.653544 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:05.653522    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-c657mlf77"]
Nov 03 11:10:06.232378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:06.232343    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9091470e-69f9-450d-978d-9ae10f0a1bbe" path="/var/lib/kubelet/pods/9091470e-69f9-450d-978d-9ae10f0a1bbe/volumes"
Nov 03 11:10:07.405059 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.405026    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"]
Nov 03 11:10:07.405780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.405765    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9091470e-69f9-450d-978d-9ae10f0a1bbe" containerName="payload-processing"
Nov 03 11:10:07.405780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.405781    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9091470e-69f9-450d-978d-9ae10f0a1bbe" containerName="payload-processing"
Nov 03 11:10:07.405955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.405945    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9091470e-69f9-450d-978d-9ae10f0a1bbe" containerName="payload-processing"
Nov 03 11:10:07.411131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.411113    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.417630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.417590    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-cbeebd-openshift-default-dockercfg-g27jf\""
Nov 03 11:10:07.418881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.418855    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"]
Nov 03 11:10:07.595531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.595489    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-podinfo\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.595730 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.595617    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.595730 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.595681    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-credential-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.595730 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.595710    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-token\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.595945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.595906    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-certs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.596027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.595998    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzp9\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-kube-api-access-sfzp9\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.596156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.596135    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-data\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.596228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.596212    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-envoy\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.596315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.596302    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istiod-ca-cert\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.697836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.697742    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-credential-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.697836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.697790    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-token\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.697885    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-certs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.697906    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzp9\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-kube-api-access-sfzp9\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698039    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-data\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698083    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-envoy\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698143    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istiod-ca-cert\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698181    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-credential-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698222    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-podinfo\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698250    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-certs\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698290    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698448    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-data\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698662    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-socket\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.698776 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.698757    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istiod-ca-cert\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.700612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.700588    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-envoy\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.700725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.700700    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-podinfo\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.707709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.707684    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzp9\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-kube-api-access-sfzp9\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.707821 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.707710    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-token\") pod \"e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") " pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:07.729456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:07.729433    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:08.020349 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:10:08.020316    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb52f8d3_88ff_4cd9_91fc_9ac95ad82c48.slice/crio-a9383e3189bf8bc5c35bc70a885c87d651bee3a4beb5ae199684fc5732191b9b WatchSource:0}: Error finding container a9383e3189bf8bc5c35bc70a885c87d651bee3a4beb5ae199684fc5732191b9b: Status 404 returned error can't find the container with id a9383e3189bf8bc5c35bc70a885c87d651bee3a4beb5ae199684fc5732191b9b
Nov 03 11:10:08.020525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:08.020501    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"]
Nov 03 11:10:08.023496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:08.023462    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:08.023595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:08.023530    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:08.023595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:08.023557    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:08.348486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:08.348453    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" event={"ID":"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48","Type":"ContainerStarted","Data":"3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae"}
Nov 03 11:10:08.348666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:08.348506    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" event={"ID":"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48","Type":"ContainerStarted","Data":"a9383e3189bf8bc5c35bc70a885c87d651bee3a4beb5ae199684fc5732191b9b"}
Nov 03 11:10:08.375167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:08.375116    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podStartSLOduration=1.375100011 podStartE2EDuration="1.375100011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:08.372444357 +0000 UTC m=+2452.277298811" watchObservedRunningTime="2026-11-03 11:10:08.375100011 +0000 UTC m=+2452.279954452"
Nov 03 11:10:08.730261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:08.730179    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:09.262154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.262115    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"]
Nov 03 11:10:09.265875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.265847    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:10:09.317944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.317683    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"]
Nov 03 11:10:09.324029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.323989    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"]
Nov 03 11:10:09.324218 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.324114    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.329365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.329184    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.329365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.329227    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.329365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.329341    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.329677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.329445    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q546r\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-kube-api-access-q546r\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.440603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.431999    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.440603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.432126    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q546r\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-kube-api-access-q546r\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.440603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.432416    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.440603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.432456    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.482943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.449116    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.482943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.462545    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.486518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.486486    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.500940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.487726    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q546r\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-kube-api-access-q546r\") pod \"payload-processing-external-model-855b7646c4-fnxx9\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.653959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.650966    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:09.765198 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.765162    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:10:09.767386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:09.767326    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 11:10:10.074003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:10.073907    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"]
Nov 03 11:10:10.079105 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:10:10.079074    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-f4e5990fecb45dee82783c2c8d228d8587229c23848eb630828eddbde506b270 WatchSource:0}: Error finding container f4e5990fecb45dee82783c2c8d228d8587229c23848eb630828eddbde506b270: Status 404 returned error can't find the container with id f4e5990fecb45dee82783c2c8d228d8587229c23848eb630828eddbde506b270
Nov 03 11:10:10.380947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:10.380283    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9" event={"ID":"b6777a1e-42f5-4685-bc68-09ae320701b4","Type":"ContainerStarted","Data":"83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06"}
Nov 03 11:10:10.380947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:10.380437    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9" event={"ID":"b6777a1e-42f5-4685-bc68-09ae320701b4","Type":"ContainerStarted","Data":"f4e5990fecb45dee82783c2c8d228d8587229c23848eb630828eddbde506b270"}
Nov 03 11:10:10.380947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:10.380496    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:10.448942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:10.448712    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9" podStartSLOduration=1.448692992 podStartE2EDuration="1.448692992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:10.445275259 +0000 UTC m=+2454.350129694" watchObservedRunningTime="2026-11-03 11:10:10.448692992 +0000 UTC m=+2454.353547433"
Nov 03 11:10:10.736999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:10.730496    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:10:10.736999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:10.730560    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 11:10:10.984940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:10.972234    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"]
Nov 03 11:10:11.732954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:11.731369    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:10:11.732954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:11.731435    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 11:10:12.073708 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.073577    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"]
Nov 03 11:10:12.078561 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.078536    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.111011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.088001    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"]
Nov 03 11:10:12.111011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.093716    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.111011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.094154    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"]
Nov 03 11:10:12.111011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.102407    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"]
Nov 03 11:10:12.158323 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.158289    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"]
Nov 03 11:10:12.172339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.172313    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"]
Nov 03 11:10:12.172497 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.172403    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.241500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.241459    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.241693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.241526    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.241693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.241650    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.241826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.241782    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.241890 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.241837    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afc9a85-df4f-4401-8656-19750d21605a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.241985    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.242173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242158    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afc9a85-df4f-4401-8656-19750d21605a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242294    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242379 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242355    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242473    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmx5g\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-kube-api-access-gmx5g\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.242556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242496    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242515    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242532    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.242713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242605    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ltk\" (UniqueName: \"kubernetes.io/projected/7afc9a85-df4f-4401-8656-19750d21605a-kube-api-access-d6ltk\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242673    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.242713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242693    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afc9a85-df4f-4401-8656-19750d21605a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242716    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.242879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.242738    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.343977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.343865    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.343977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.343965    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.343997    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afc9a85-df4f-4401-8656-19750d21605a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344024    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344058    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afc9a85-df4f-4401-8656-19750d21605a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344089    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344121    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344181    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344205    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344257    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.344297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344296    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmx5g\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-kube-api-access-gmx5g\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.344763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344312    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.344763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344326    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.344763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344342    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.344763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344623    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.344763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344644    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.345047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344880    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.345047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344910    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.345047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.344951    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.345215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345082    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858zx\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-kube-api-access-858zx\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.345215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345107    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ltk\" (UniqueName: \"kubernetes.io/projected/7afc9a85-df4f-4401-8656-19750d21605a-kube-api-access-d6ltk\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.345215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345135    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.345215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345152    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afc9a85-df4f-4401-8656-19750d21605a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.345215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345177    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.345215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345199    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.345521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345223    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.345521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345257    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.345521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345273    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.345521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345293    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef6c43b-90bb-4158-8036-1c3b70f16396-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.345521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345333    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.345521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345371    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.345898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345877    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.346009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.345986    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.346251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.346228    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.346321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.346259    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afc9a85-df4f-4401-8656-19750d21605a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.346737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.346699    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.347578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.347553    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.347725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.347702    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afc9a85-df4f-4401-8656-19750d21605a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.347838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.347795    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.349123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.349097    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afc9a85-df4f-4401-8656-19750d21605a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.349521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.349501    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.354832 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.354812    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmx5g\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-kube-api-access-gmx5g\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.355426 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.355405    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ltk\" (UniqueName: \"kubernetes.io/projected/7afc9a85-df4f-4401-8656-19750d21605a-kube-api-access-d6ltk\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.356514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.356489    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afc9a85-df4f-4401-8656-19750d21605a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-hs9mc\" (UID: \"7afc9a85-df4f-4401-8656-19750d21605a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.357022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.357002    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fwgj9\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.410421 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.410389    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:12.446130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.446098    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:12.447092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.447065    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.447435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.447421    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.447555 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.447543    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.447969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.447878    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.448168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.448153    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.448560 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.448546    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.448980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.448861    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.449086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.448908    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858zx\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-kube-api-access-858zx\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.449225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.449213    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.449308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.449298    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.449383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.449373    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef6c43b-90bb-4158-8036-1c3b70f16396-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.449574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.449553    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.449763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.449729    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.449825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.449757    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.450382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.450366    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef6c43b-90bb-4158-8036-1c3b70f16396-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.450947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.450894    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.463970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.463808    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858zx\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-kube-api-access-858zx\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.464124 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.463999    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fcrct\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.525462 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.525422    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:12.749232 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.748405    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:10:12.749232 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:12.748471    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 11:10:13.128910 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:10:13.127795    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d98f5bc_ccdf_4f23_b64d_9612d31d89e2.slice/crio-39a6c78340efaf7e4abc77f6e4afa049e8e73a6c368a8fe67c9bf3e4040d6a45 WatchSource:0}: Error finding container 39a6c78340efaf7e4abc77f6e4afa049e8e73a6c368a8fe67c9bf3e4040d6a45: Status 404 returned error can't find the container with id 39a6c78340efaf7e4abc77f6e4afa049e8e73a6c368a8fe67c9bf3e4040d6a45
Nov 03 11:10:13.141312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.133868    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"]
Nov 03 11:10:13.141312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.133992    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"]
Nov 03 11:10:13.148327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.147359    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.148327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.147442    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.148327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.147509    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.148327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.147576    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.148327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.147632    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.148327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.147656    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.285492 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:13.281418    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:10:13.329302 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:10:13.328175    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef6c43b_90bb_4158_8036_1c3b70f16396.slice/crio-0a0cc2fdfaf3c305a88f49fe39ca1dab8efc66decbeedb08210f07efa4438ba9 WatchSource:0}: Error finding container 0a0cc2fdfaf3c305a88f49fe39ca1dab8efc66decbeedb08210f07efa4438ba9: Status 404 returned error can't find the container with id 0a0cc2fdfaf3c305a88f49fe39ca1dab8efc66decbeedb08210f07efa4438ba9
Nov 03 11:10:13.330378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.330340    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"]
Nov 03 11:10:13.334873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.334843    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.334968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.334906    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.335026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.334965    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:10:13.449398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.449362    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" event={"ID":"4ef6c43b-90bb-4158-8036-1c3b70f16396","Type":"ContainerStarted","Data":"0a0cc2fdfaf3c305a88f49fe39ca1dab8efc66decbeedb08210f07efa4438ba9"}
Nov 03 11:10:13.452081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.452046    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" event={"ID":"7afc9a85-df4f-4401-8656-19750d21605a","Type":"ContainerStarted","Data":"1e95709b1ca67a36602cbef500f512d29785f2c48067792ce2aed81320d64ed4"}
Nov 03 11:10:13.452222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.452105    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" event={"ID":"7afc9a85-df4f-4401-8656-19750d21605a","Type":"ContainerStarted","Data":"ab8bde9de9de17b3d252101a8bf3c57ddf13730e31230858c030b099d76e14d5"}
Nov 03 11:10:13.463802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.463766    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" event={"ID":"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2","Type":"ContainerStarted","Data":"1be0b38f6ff11d4c06abbf70677466ebdb3ded495626c50df47ba4203a1b19b3"}
Nov 03 11:10:13.463970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.463837    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" event={"ID":"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2","Type":"ContainerStarted","Data":"39a6c78340efaf7e4abc77f6e4afa049e8e73a6c368a8fe67c9bf3e4040d6a45"}
Nov 03 11:10:13.479897 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.479822    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podStartSLOduration=1.479802472 podStartE2EDuration="1.479802472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:13.475974334 +0000 UTC m=+2457.380828776" watchObservedRunningTime="2026-11-03 11:10:13.479802472 +0000 UTC m=+2457.384656913"
Nov 03 11:10:13.534854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.534744    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podStartSLOduration=1.5347230330000001 podStartE2EDuration="1.534723033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:13.527535502 +0000 UTC m=+2457.432389955" watchObservedRunningTime="2026-11-03 11:10:13.534723033 +0000 UTC m=+2457.439577476"
Nov 03 11:10:13.747108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.747017    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:10:13.747437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:13.747394    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 11:10:14.277597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.270193    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf" podUID="536d109b-d30e-49e4-aed0-59eb676c5830" containerName="payload-processing" containerID="cri-o://5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014" gracePeriod=25
Nov 03 11:10:14.501936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.501882    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:14.505782 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.505751    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:14.506077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.506051    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:14.562886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.562803    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:14.697643 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.629422    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"]
Nov 03 11:10:14.730780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.730358    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:10:14.730780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.730414    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 11:10:14.856755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.811779    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"]
Nov 03 11:10:14.858415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.830359    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:14.893256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.834580    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:14.893256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.865323    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:14.893256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.865578    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:14.893256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.865668    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6ql\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-kube-api-access-7h6ql\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:14.893256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.875451    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"payload-processing-external-model-e2e--90f9edc7-dockercfg-sf82p\""
Nov 03 11:10:14.909002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.900557    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-917da2\""
Nov 03 11:10:14.929992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.909466    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"openshift-service-ca.crt\""
Nov 03 11:10:14.929992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.880255    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"kube-root-ca.crt\""
Nov 03 11:10:14.978757 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.968880    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"agc-cred-2cc0333784c3\""
Nov 03 11:10:14.978757 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.969081    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:14.978757 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:14.969145    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:15.016039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.006953    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"routing-overlay\""
Nov 03 11:10:15.078668 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:15.078625    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-conmon-5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:15.153239 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.092389    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:15.153397 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.145383    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.153598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.153569    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.153875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.153850    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6ql\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-kube-api-access-7h6ql\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.154293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.154267    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.171530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.171486    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.172411 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.172341    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.196995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.196158    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" event={"ID":"4ef6c43b-90bb-4158-8036-1c3b70f16396","Type":"ContainerStarted","Data":"cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc"}
Nov 03 11:10:15.198521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.198496    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6ql\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-kube-api-access-7h6ql\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.218896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.218829    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.248339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.248186    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podStartSLOduration=3.248168583 podStartE2EDuration="3.248168583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:15.243120268 +0000 UTC m=+2459.147974709" watchObservedRunningTime="2026-11-03 11:10:15.248168583 +0000 UTC m=+2459.153023025"
Nov 03 11:10:15.449601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.449498    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:15.450286 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.450256    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:15.450819 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.450195    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:15.451026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.450997    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:15.469714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.469636    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:15.574947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.564441    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:15.602741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.602424    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:15.602741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.602478    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:15.736020 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.735511    5393 patch_prober.go:29] interesting pod/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 11:10:15.736020 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.735574    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 11:10:15.738038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.737653    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:10:15.794752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.793178    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-plugins-config-volume\") pod \"536d109b-d30e-49e4-aed0-59eb676c5830\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") "
Nov 03 11:10:15.794752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.793240    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-kube-api-access-jdcnn\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-kube-api-access-jdcnn\") pod \"536d109b-d30e-49e4-aed0-59eb676c5830\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") "
Nov 03 11:10:15.794752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.793416    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-provider-credentials\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-provider-credentials\") pod \"536d109b-d30e-49e4-aed0-59eb676c5830\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") "
Nov 03 11:10:15.794752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.793454    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-routing-overlay-volume\") pod \"536d109b-d30e-49e4-aed0-59eb676c5830\" (UID: \"536d109b-d30e-49e4-aed0-59eb676c5830\") "
Nov 03 11:10:15.795912 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.795579    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-plugins-config-volume" pod "536d109b-d30e-49e4-aed0-59eb676c5830" (UID: "536d109b-d30e-49e4-aed0-59eb676c5830"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:15.835383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.827256    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-routing-overlay-volume" pod "536d109b-d30e-49e4-aed0-59eb676c5830" (UID: "536d109b-d30e-49e4-aed0-59eb676c5830"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:15.846158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.846098    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-kube-api-access-jdcnn" pod "536d109b-d30e-49e4-aed0-59eb676c5830" (UID: "536d109b-d30e-49e4-aed0-59eb676c5830"). InnerVolumeSpecName "kube-api-access-jdcnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:15.981011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:15.924697    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:16.018943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.018542    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcnn\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-kube-api-access-jdcnn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:16.018943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.018590    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/536d109b-d30e-49e4-aed0-59eb676c5830-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:16.146550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.146490    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-provider-credentials" pod "536d109b-d30e-49e4-aed0-59eb676c5830" (UID: "536d109b-d30e-49e4-aed0-59eb676c5830"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:16.147387 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.147326    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/536d109b-d30e-49e4-aed0-59eb676c5830-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:16.641951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.631682    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:16.641951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.631749    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:16.984944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.984430    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 03 11:10:16.984944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.984850    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:17.054941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.027331    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:17.085676 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:16.987906    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:17.099078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.099030    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:17.134042 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.133995    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:17.155943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.134303    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:17.156384 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.156342    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:17.165062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.165003    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:17.217131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.179202    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:17.399010 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.281714    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 11:10:17.461348 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.432875    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9" podUID="b6777a1e-42f5-4685-bc68-09ae320701b4" containerName="payload-processing" containerID="cri-o://83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06" gracePeriod=25
Nov 03 11:10:17.697886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.697788    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:17.698496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.698450    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:17.699885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.699853    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:17.700101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.700081    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:17.730418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.712348    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:17.730418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.712402    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:17.730418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.719250    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:17.730418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.719301    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:17.816032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.806000    5393 generic.go:356] "Generic (PLEG): container finished" podID="536d109b-d30e-49e4-aed0-59eb676c5830" containerID="5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014" exitCode=0
Nov 03 11:10:17.816032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.811779    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"
Nov 03 11:10:17.920950 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:17.911771    5393 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.425s"
Nov 03 11:10:17.941953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.939153    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf" event={"ID":"536d109b-d30e-49e4-aed0-59eb676c5830","Type":"ContainerDied","Data":"5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014"}
Nov 03 11:10:17.941953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.939233    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf" event={"ID":"536d109b-d30e-49e4-aed0-59eb676c5830","Type":"ContainerDied","Data":"b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5"}
Nov 03 11:10:17.941953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.939354    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:17.941953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.939385    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"]
Nov 03 11:10:17.941953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:17.941151    5393 scope.go:122] "RemoveContainer" containerID="5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014"
Nov 03 11:10:18.151945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.148656    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:18.287008 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.267877    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:10:18.556231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.552372    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:18.556231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.552439    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:18.556231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.552537    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:18.556231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.552570    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:18.638546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.608727    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:18.638546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.608792    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:18.663850 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.663823    5393 scope.go:122] "RemoveContainer" containerID="5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014"
Nov 03 11:10:18.664583 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:18.664559    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014\": container with ID starting with 5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014 not found: ID does not exist" containerID="5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014"
Nov 03 11:10:18.664765 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.664726    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014"} err="failed to get container status \"5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014\": rpc error: code = NotFound desc = could not find container \"5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014\": container with ID starting with 5bb6df9dc937d019c7549cae732a5a1893d36ab6b5a0f6d55be8a7e665985014 not found: ID does not exist"
Nov 03 11:10:18.682538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.678043    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:18.682538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.678572    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:18.683539 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.683504    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:18.689968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.683691    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:18.689968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.683734    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 11:10:18.716942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.716468    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:18.717435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.717087    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 11:10:18.736641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.727696    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 11:10:18.736641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.730462    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded"
Nov 03 11:10:18.736641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.731529    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 11:10:18.799351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.799317    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 11:10:18.799629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.799615    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918363    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918703    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918744    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"]
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918770    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8444774d4b-dvzrf"]
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918813    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918850    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918886    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918949    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.918986    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 11:10:18.920789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.919012    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:10:18.939663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:18.924768    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 11:10:19.027079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.027034    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 11:10:19.455587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.455493    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:19.455850 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.455812    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:19.527428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.527394    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:19.527694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.527670    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:19.530405 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.530375    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:19.564861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.564813    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:19.592491 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.592451    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf" event={"ID":"fc68a473-21f6-4b51-ae17-bbd1a5effcc4","Type":"ContainerStarted","Data":"c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f"}
Nov 03 11:10:19.593565 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.593530    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf" event={"ID":"fc68a473-21f6-4b51-ae17-bbd1a5effcc4","Type":"ContainerStarted","Data":"e3e7a384cc2cdc3c6414843c48c06ee7a9b03814ef55b9b85e0cdac9db98d0ec"}
Nov 03 11:10:19.596559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.596533    5393 generic.go:356] "Generic (PLEG): container finished" podID="b6777a1e-42f5-4685-bc68-09ae320701b4" containerID="83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06" exitCode=0
Nov 03 11:10:19.597787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.597764    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9" event={"ID":"b6777a1e-42f5-4685-bc68-09ae320701b4","Type":"ContainerDied","Data":"83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06"}
Nov 03 11:10:19.598026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.598010    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:19.701424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.701387    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 11:10:19.882583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.869562    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 11:10:19.919799 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.919763    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 11:10:19.921022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.920989    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 11:10:19.921652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.921621    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 11:10:19.979004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:19.948667    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf" podStartSLOduration=5.948645465 podStartE2EDuration="5.948645465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:19.633335185 +0000 UTC m=+2463.538189626" watchObservedRunningTime="2026-11-03 11:10:19.948645465 +0000 UTC m=+2463.853499907"
Nov 03 11:10:20.046001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.037534    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 11:10:20.046001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.038189    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 11:10:20.101945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.075216    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:20.163946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.142093    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-kube-api-access-q546r\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-kube-api-access-q546r\") pod \"b6777a1e-42f5-4685-bc68-09ae320701b4\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") "
Nov 03 11:10:20.321142 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.321045    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-plugins-config-volume\") pod \"b6777a1e-42f5-4685-bc68-09ae320701b4\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") "
Nov 03 11:10:20.344944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.325058    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-routing-overlay-volume\") pod \"b6777a1e-42f5-4685-bc68-09ae320701b4\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") "
Nov 03 11:10:20.344944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.325315    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-provider-credentials\") pod \"b6777a1e-42f5-4685-bc68-09ae320701b4\" (UID: \"b6777a1e-42f5-4685-bc68-09ae320701b4\") "
Nov 03 11:10:20.344944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.340037    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-routing-overlay-volume" pod "b6777a1e-42f5-4685-bc68-09ae320701b4" (UID: "b6777a1e-42f5-4685-bc68-09ae320701b4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:20.419178 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.417415    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:20.419178 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.417472    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:20.481018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.419599    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-plugins-config-volume" pod "b6777a1e-42f5-4685-bc68-09ae320701b4" (UID: "b6777a1e-42f5-4685-bc68-09ae320701b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:20.498532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.445638    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:20.498728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.421105    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-kube-api-access-q546r" pod "b6777a1e-42f5-4685-bc68-09ae320701b4" (UID: "b6777a1e-42f5-4685-bc68-09ae320701b4"). InnerVolumeSpecName "kube-api-access-q546r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:20.543077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.543043    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:20.543321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.543292    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:20.543873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.543847    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:20.558021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.557985    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:20.647987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.596047    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-provider-credentials" pod "b6777a1e-42f5-4685-bc68-09ae320701b4" (UID: "b6777a1e-42f5-4685-bc68-09ae320701b4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:20.647987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.631625    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536d109b-d30e-49e4-aed0-59eb676c5830" path="/var/lib/kubelet/pods/536d109b-d30e-49e4-aed0-59eb676c5830/volumes"
Nov 03 11:10:20.665967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.665911    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:20.722192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.708397    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q546r\" (UniqueName: \"kubernetes.io/projected/b6777a1e-42f5-4685-bc68-09ae320701b4-kube-api-access-q546r\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:20.722192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.708431    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6777a1e-42f5-4685-bc68-09ae320701b4-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:20.722192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.703790    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 11:10:20.829084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.829052    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"
Nov 03 11:10:20.837299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.837257    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9" event={"ID":"b6777a1e-42f5-4685-bc68-09ae320701b4","Type":"ContainerDied","Data":"f4e5990fecb45dee82783c2c8d228d8587229c23848eb630828eddbde506b270"}
Nov 03 11:10:20.837556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.837535    5393 scope.go:122] "RemoveContainer" containerID="83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06"
Nov 03 11:10:20.875362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.875223    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 11:10:20.942211 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.942120    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:20.944278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.942733    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:20.975101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.974983    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:20.976115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:20.976074    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:21.033946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.018077    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"]
Nov 03 11:10:21.039213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.039167    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:21.070478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.070238    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:21.091944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.091243    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-fnxx9"]
Nov 03 11:10:21.127306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.127162    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:21.307997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.301022    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 11:10:21.539768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.539613    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:21.540155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.540108    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:21.541108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.540909    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:21.542815 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.542789    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:21.566742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.566650    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:21.567691 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.567659    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:21.924972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.924329    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 11:10:21.953948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.948797    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:21.990944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.968005    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:21.990944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.988273    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:21.990944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:21.988872    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:22.043062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.043011    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:22.052526 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.052489    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"]
Nov 03 11:10:22.054051 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.054022    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b6777a1e-42f5-4685-bc68-09ae320701b4" containerName="payload-processing"
Nov 03 11:10:22.056025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.056000    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6777a1e-42f5-4685-bc68-09ae320701b4" containerName="payload-processing"
Nov 03 11:10:22.056281 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.056266    5393 cpu_manager.go:414] "containerMap: removing container" podUID="536d109b-d30e-49e4-aed0-59eb676c5830" containerName="payload-processing"
Nov 03 11:10:22.056401 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.056389    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536d109b-d30e-49e4-aed0-59eb676c5830" containerName="payload-processing"
Nov 03 11:10:22.056724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.056705    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="536d109b-d30e-49e4-aed0-59eb676c5830" containerName="payload-processing"
Nov 03 11:10:22.056838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.056827    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6777a1e-42f5-4685-bc68-09ae320701b4" containerName="payload-processing"
Nov 03 11:10:22.112171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.075452    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"]
Nov 03 11:10:22.112171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.075578    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.112171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.104339    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-abf28fc53a90\""
Nov 03 11:10:22.115670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.076902    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:22.127365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.127324    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 11:10:22.153390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.143437    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 11:10:22.180137 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.180045    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:10:22.228209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.222743    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x952\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-kube-api-access-4x952\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.465340 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.465250    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:22.465647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.465614    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:22.502072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.502028    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:22.503248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.503211    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:22.508253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.508203    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.508656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.508630    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.508862 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.508768    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.697945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.665633    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:22.697945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.665696    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:22.745013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.713540    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6777a1e-42f5-4685-bc68-09ae320701b4" path="/var/lib/kubelet/pods/b6777a1e-42f5-4685-bc68-09ae320701b4/volumes"
Nov 03 11:10:22.966772 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.966719    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:22.971839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.971680    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.972690 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.972658    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.978134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.978106    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x952\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-kube-api-access-4x952\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.988668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.988617    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:22.994827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.990868    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-plugins-config-volume\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:23.002987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:22.978535    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:23.002987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.001201    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 11:10:23.002987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.001288    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 11:10:23.037346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.031977    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:23.054978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.051393    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 11:10:23.054978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.053386    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:10:23.054978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.054508    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 11:10:23.072942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.070484    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-routing-overlay-volume\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:23.076943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.073479    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-provider-credentials\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:23.076943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.076909    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 11:10:23.107068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.094843    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 11:10:23.259272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.259149    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:23.260799 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.260763    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 11:10:23.383997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.382026    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x952\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-kube-api-access-4x952\") pod \"payload-processing-external-model-855b7646c4-2wc6s\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") " pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:23.393376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.393343    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:23.459869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.459837    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:23.460278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.460226    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:23.568605 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.568487    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:23.569164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.569123    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:23.569438 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.569408    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:23.569619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:23.569586    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:24.243252 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.225548    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded"
Nov 03 11:10:24.282986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.273709    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:24.431955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.423203    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:24.454172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.454134    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:24.464270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.450484    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:24.464562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.464535    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:24.473175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.473094    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"]
Nov 03 11:10:24.487603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.483776    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"]
Nov 03 11:10:24.530060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.526687    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:24.530060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.526767    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:24.545943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.544436    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"]
Nov 03 11:10:24.545943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.544569    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.558023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.547585    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-provider-credentials\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.558023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.547648    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-plugins-config-volume\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.558023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.547734    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdf8\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-kube-api-access-shdf8\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.558023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.547978    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-routing-overlay-volume\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.558023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.556374    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a38b0871f9a0\""
Nov 03 11:10:24.658266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.656278    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-routing-overlay-volume\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.658266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.656384    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-provider-credentials\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.658266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.656428    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-plugins-config-volume\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.658266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.656494    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdf8\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-kube-api-access-shdf8\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.658266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.657820    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-routing-overlay-volume\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.663281 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.663240    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-plugins-config-volume\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.757198 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.755187    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 11:10:24.824079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.801133    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-provider-credentials\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.854087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.852646    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdf8\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-kube-api-access-shdf8\") pod \"payload-processing-external-model-5f57c946bf-9gvjt\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") " pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:24.864621 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.863524    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 11:10:24.900568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:24.900292    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:25.247622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:25.247569    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"]
Nov 03 11:10:25.448027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:25.441147    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:25.448027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:25.441205    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:25.463208 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:25.462346    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:25.463208 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:25.462403    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:25.463731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:25.463702    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"]
Nov 03 11:10:25.592432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:25.582520    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:25.592432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:25.582576    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:26.011143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:26.011109    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s" event={"ID":"8e207249-2b8c-4253-9274-a54afc2d0412","Type":"ContainerStarted","Data":"979475ab0685cf4c8319b9226feb0a4d92242d53990befdd90db56c9e0e558d6"}
Nov 03 11:10:26.050307 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:26.050266    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:26.261808 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:10:26.261762    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96462446_12de_4613_ab2c_526f09a14870.slice/crio-ce57b202d8a96a3df876dc6b9a574265e1891fc6321232d76fdda2f894f800fa WatchSource:0}: Error finding container ce57b202d8a96a3df876dc6b9a574265e1891fc6321232d76fdda2f894f800fa: Status 404 returned error can't find the container with id ce57b202d8a96a3df876dc6b9a574265e1891fc6321232d76fdda2f894f800fa
Nov 03 11:10:26.265192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:26.265127    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"]
Nov 03 11:10:26.412728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:26.412291    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:26.412728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:26.412350    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:26.447425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:26.447388    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:26.447591 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:26.447453    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:26.526442 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:26.526366    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:26.526581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:26.526432    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:27.026344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.026301    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt" event={"ID":"96462446-12de-4613-ab2c-526f09a14870","Type":"ContainerStarted","Data":"8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18"}
Nov 03 11:10:27.026548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.026413    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt" event={"ID":"96462446-12de-4613-ab2c-526f09a14870","Type":"ContainerStarted","Data":"ce57b202d8a96a3df876dc6b9a574265e1891fc6321232d76fdda2f894f800fa"}
Nov 03 11:10:27.026548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.026449    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:27.028584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.028542    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s" event={"ID":"8e207249-2b8c-4253-9274-a54afc2d0412","Type":"ContainerStarted","Data":"69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2"}
Nov 03 11:10:27.028774 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.028681    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:27.049504 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.049442    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt" podStartSLOduration=3.049424785 podStartE2EDuration="3.049424785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:27.044252464 +0000 UTC m=+2470.949106907" watchObservedRunningTime="2026-11-03 11:10:27.049424785 +0000 UTC m=+2470.954279227"
Nov 03 11:10:27.068113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.068043    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s" podStartSLOduration=6.068025723 podStartE2EDuration="6.068025723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:27.065531117 +0000 UTC m=+2470.970385559" watchObservedRunningTime="2026-11-03 11:10:27.068025723 +0000 UTC m=+2470.972880165"
Nov 03 11:10:27.411603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.411566    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:27.412187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.411830    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Nov 03 11:10:27.412187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.411886    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Nov 03 11:10:27.446466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.446417    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:27.446806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.446778    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body=
Nov 03 11:10:27.446951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.446841    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused"
Nov 03 11:10:27.526138 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.526104    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:27.526436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.526402    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 11:10:27.526578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:27.526463    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 11:10:28.721944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:28.714075    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:28.721944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:28.718149    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:10:29.081497 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:29.080391    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:29.082318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:29.082295    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc"
Nov 03 11:10:29.404626 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:29.404581    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:29.437399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:29.437367    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:29.440482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:29.440455    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:10:32.029945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.029348    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s" podUID="8e207249-2b8c-4253-9274-a54afc2d0412" containerName="payload-processing" containerID="cri-o://69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2" gracePeriod=25
Nov 03 11:10:32.029945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.029513    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt" podUID="96462446-12de-4613-ab2c-526f09a14870" containerName="payload-processing" containerID="cri-o://8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18" gracePeriod=25
Nov 03 11:10:32.056796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.044176    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:10:32.341172 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:32.341088    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96462446_12de_4613_ab2c_526f09a14870.slice/crio-conmon-8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e207249_2b8c_4253_9274_a54afc2d0412.slice/crio-conmon-69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:32.343884 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:32.343810    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e207249_2b8c_4253_9274_a54afc2d0412.slice/crio-conmon-69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96462446_12de_4613_ab2c_526f09a14870.slice/crio-conmon-8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:32.586843 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.586803    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:32.694945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.693827    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-kube-api-access-4x952\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-kube-api-access-4x952\") pod \"8e207249-2b8c-4253-9274-a54afc2d0412\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") "
Nov 03 11:10:32.694945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.694060    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-provider-credentials\") pod \"8e207249-2b8c-4253-9274-a54afc2d0412\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") "
Nov 03 11:10:32.694945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.694107    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-plugins-config-volume\") pod \"8e207249-2b8c-4253-9274-a54afc2d0412\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") "
Nov 03 11:10:32.694945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.694321    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-routing-overlay-volume\") pod \"8e207249-2b8c-4253-9274-a54afc2d0412\" (UID: \"8e207249-2b8c-4253-9274-a54afc2d0412\") "
Nov 03 11:10:32.703002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.695961    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-routing-overlay-volume" pod "8e207249-2b8c-4253-9274-a54afc2d0412" (UID: "8e207249-2b8c-4253-9274-a54afc2d0412"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:32.703002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.696341    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-plugins-config-volume" pod "8e207249-2b8c-4253-9274-a54afc2d0412" (UID: "8e207249-2b8c-4253-9274-a54afc2d0412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:32.705464 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.705431    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-provider-credentials" pod "8e207249-2b8c-4253-9274-a54afc2d0412" (UID: "8e207249-2b8c-4253-9274-a54afc2d0412"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:32.705618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.705528    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-kube-api-access-4x952" pod "8e207249-2b8c-4253-9274-a54afc2d0412" (UID: "8e207249-2b8c-4253-9274-a54afc2d0412"). InnerVolumeSpecName "kube-api-access-4x952". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:32.748292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.747837    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:32.797730 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.797639    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x952\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-kube-api-access-4x952\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:32.798020 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.797999    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e207249-2b8c-4253-9274-a54afc2d0412-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:32.798159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.798142    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:32.798259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.798244    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e207249-2b8c-4253-9274-a54afc2d0412-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:32.898875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.898846    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-plugins-config-volume\") pod \"96462446-12de-4613-ab2c-526f09a14870\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") "
Nov 03 11:10:32.898875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.898881    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-kube-api-access-shdf8\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-kube-api-access-shdf8\") pod \"96462446-12de-4613-ab2c-526f09a14870\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") "
Nov 03 11:10:32.899112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.898987    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-routing-overlay-volume\") pod \"96462446-12de-4613-ab2c-526f09a14870\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") "
Nov 03 11:10:32.899228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.899207    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-provider-credentials\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-provider-credentials\") pod \"96462446-12de-4613-ab2c-526f09a14870\" (UID: \"96462446-12de-4613-ab2c-526f09a14870\") "
Nov 03 11:10:32.899462 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.899395    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-plugins-config-volume" pod "96462446-12de-4613-ab2c-526f09a14870" (UID: "96462446-12de-4613-ab2c-526f09a14870"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:32.899462 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.899427    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-routing-overlay-volume" pod "96462446-12de-4613-ab2c-526f09a14870" (UID: "96462446-12de-4613-ab2c-526f09a14870"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:32.900122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.900095    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:32.900406 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.900126    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96462446-12de-4613-ab2c-526f09a14870-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:32.901844 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.901816    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-provider-credentials" pod "96462446-12de-4613-ab2c-526f09a14870" (UID: "96462446-12de-4613-ab2c-526f09a14870"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:32.901975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:32.901844    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-kube-api-access-shdf8" pod "96462446-12de-4613-ab2c-526f09a14870" (UID: "96462446-12de-4613-ab2c-526f09a14870"). InnerVolumeSpecName "kube-api-access-shdf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:33.002122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.002092    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:33.002122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.002120    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shdf8\" (UniqueName: \"kubernetes.io/projected/96462446-12de-4613-ab2c-526f09a14870-kube-api-access-shdf8\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:33.174525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.174460    5393 generic.go:356] "Generic (PLEG): container finished" podID="8e207249-2b8c-4253-9274-a54afc2d0412" containerID="69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2" exitCode=0
Nov 03 11:10:33.174525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.174514    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"
Nov 03 11:10:33.175105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.174551    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s" event={"ID":"8e207249-2b8c-4253-9274-a54afc2d0412","Type":"ContainerDied","Data":"69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2"}
Nov 03 11:10:33.175105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.174611    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s" event={"ID":"8e207249-2b8c-4253-9274-a54afc2d0412","Type":"ContainerDied","Data":"979475ab0685cf4c8319b9226feb0a4d92242d53990befdd90db56c9e0e558d6"}
Nov 03 11:10:33.175105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.174636    5393 scope.go:122] "RemoveContainer" containerID="69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2"
Nov 03 11:10:33.176295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.176274    5393 generic.go:356] "Generic (PLEG): container finished" podID="96462446-12de-4613-ab2c-526f09a14870" containerID="8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18" exitCode=0
Nov 03 11:10:33.176380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.176320    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"
Nov 03 11:10:33.176380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.176360    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt" event={"ID":"96462446-12de-4613-ab2c-526f09a14870","Type":"ContainerDied","Data":"8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18"}
Nov 03 11:10:33.176449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.176404    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt" event={"ID":"96462446-12de-4613-ab2c-526f09a14870","Type":"ContainerDied","Data":"ce57b202d8a96a3df876dc6b9a574265e1891fc6321232d76fdda2f894f800fa"}
Nov 03 11:10:33.186742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.186724    5393 scope.go:122] "RemoveContainer" containerID="69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2"
Nov 03 11:10:33.187238 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:33.187217    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2\": container with ID starting with 69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2 not found: ID does not exist" containerID="69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2"
Nov 03 11:10:33.187330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.187246    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2"} err="failed to get container status \"69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2\": rpc error: code = NotFound desc = could not find container \"69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2\": container with ID starting with 69357da9aaba3de8f394f1f48f105c4310a4e31c5cb6403c20402db1a6a267e2 not found: ID does not exist"
Nov 03 11:10:33.187330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.187262    5393 scope.go:122] "RemoveContainer" containerID="8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18"
Nov 03 11:10:33.197202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.197183    5393 scope.go:122] "RemoveContainer" containerID="8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18"
Nov 03 11:10:33.197466 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:33.197444    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18\": container with ID starting with 8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18 not found: ID does not exist" containerID="8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18"
Nov 03 11:10:33.197517 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.197473    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18"} err="failed to get container status \"8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18\": rpc error: code = NotFound desc = could not find container \"8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18\": container with ID starting with 8524aa0d08f11dbd7de5d8ebc9f22f2e568186c446aa599641ead9ec8a743d18 not found: ID does not exist"
Nov 03 11:10:33.206149 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.206124    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"]
Nov 03 11:10:33.210196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.210174    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-855b7646c4-2wc6s"]
Nov 03 11:10:33.219904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.219872    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"]
Nov 03 11:10:33.225380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:33.225356    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f57c946bf-9gvjt"]
Nov 03 11:10:34.232134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:34.232104    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e207249-2b8c-4253-9274-a54afc2d0412" path="/var/lib/kubelet/pods/8e207249-2b8c-4253-9274-a54afc2d0412/volumes"
Nov 03 11:10:34.232571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:34.232557    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96462446-12de-4613-ab2c-526f09a14870" path="/var/lib/kubelet/pods/96462446-12de-4613-ab2c-526f09a14870/volumes"
Nov 03 11:10:36.113263 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:36.113221    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:40.306047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.306014    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"]
Nov 03 11:10:40.308299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.308275    5393 cpu_manager.go:414] "containerMap: removing container" podUID="8e207249-2b8c-4253-9274-a54afc2d0412" containerName="payload-processing"
Nov 03 11:10:40.308299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.308299    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e207249-2b8c-4253-9274-a54afc2d0412" containerName="payload-processing"
Nov 03 11:10:40.308506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.308339    5393 cpu_manager.go:414] "containerMap: removing container" podUID="96462446-12de-4613-ab2c-526f09a14870" containerName="payload-processing"
Nov 03 11:10:40.308506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.308346    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96462446-12de-4613-ab2c-526f09a14870" containerName="payload-processing"
Nov 03 11:10:40.308732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.308716    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e207249-2b8c-4253-9274-a54afc2d0412" containerName="payload-processing"
Nov 03 11:10:40.308810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.308741    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="96462446-12de-4613-ab2c-526f09a14870" containerName="payload-processing"
Nov 03 11:10:40.321968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.321912    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"]
Nov 03 11:10:40.322209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.322087    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.328951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.328876    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5711cf\""
Nov 03 11:10:40.329103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.328995    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"routing-overlay\""
Nov 03 11:10:40.329103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.329098    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"kube-root-ca.crt\""
Nov 03 11:10:40.329220 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.329155    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"openshift-service-ca.crt\""
Nov 03 11:10:40.345482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.342212    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"agc-cred-b8c3e972a39f\""
Nov 03 11:10:40.489039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.483028    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.489039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.483163    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.489039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.483491    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.489039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.483575    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b76wf\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-kube-api-access-b76wf\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.620996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.596024    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.620996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.620263    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.620996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.620360    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b76wf\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-kube-api-access-b76wf\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.620996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.620554    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.622377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.622340    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.623175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.623138    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:40.954398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:40.954302    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"]
Nov 03 11:10:41.196947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.177969    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b76wf\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-kube-api-access-b76wf\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:41.199285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.198957    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:41.254596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.254505    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:41.499726 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.499695    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"]
Nov 03 11:10:41.543157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.500322    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.600798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.570143    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"routing-overlay\""
Nov 03 11:10:41.600798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.587283    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"openshift-service-ca.crt\""
Nov 03 11:10:41.600798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.587364    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9f53bb\""
Nov 03 11:10:41.600798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.587648    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"kube-root-ca.crt\""
Nov 03 11:10:41.724291 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.724244    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.727940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.724656    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.809603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.742562    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:41.811276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.811241    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.812384 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.812357    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfq98\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-kube-api-access-cfq98\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.840952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.782341    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9f53bb\"/\"agc-cred-4cb90d41b019\""
Nov 03 11:10:41.935087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.925372    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.935087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.925455    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfq98\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-kube-api-access-cfq98\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.935087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.925562    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.935087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.925648    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.935087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.926692    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.935087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.926766    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:41.949028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.936332    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:41.949028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.937674    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:41.970059 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:41.970013    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:42.022368 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.010709    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfq98\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-kube-api-access-cfq98\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:42.022368 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.010821    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:42.044077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.029061    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 11:10:42.113543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.113509    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") " pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:42.135039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.134995    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 03 11:10:42.136506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.136474    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded"
Nov 03 11:10:42.267642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.241291    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:42.310813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.306784    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:42.370040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.369523    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:42.678942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.677533    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"]
Nov 03 11:10:42.689030 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:10:42.689003    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9038b83_5c62_4a08_9a21_1333829d10b6.slice/crio-77d1106987eaf3a2dba42d71a465795b83d639aa72d66685ec6bec7cdd2c0a6b WatchSource:0}: Error finding container 77d1106987eaf3a2dba42d71a465795b83d639aa72d66685ec6bec7cdd2c0a6b: Status 404 returned error can't find the container with id 77d1106987eaf3a2dba42d71a465795b83d639aa72d66685ec6bec7cdd2c0a6b
Nov 03 11:10:42.900415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.900381    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"]
Nov 03 11:10:42.901094 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:10:42.901065    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc05d406_c2f5_489f_81e9_c9b32339b6e8.slice/crio-c9f7e6b6ccd45cd47a1bf273f5fc307db55196fa3c6a1b75c0e24d25f6d7420f WatchSource:0}: Error finding container c9f7e6b6ccd45cd47a1bf273f5fc307db55196fa3c6a1b75c0e24d25f6d7420f: Status 404 returned error can't find the container with id c9f7e6b6ccd45cd47a1bf273f5fc307db55196fa3c6a1b75c0e24d25f6d7420f
Nov 03 11:10:42.996702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.996668    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd" event={"ID":"c9038b83-5c62-4a08-9a21-1333829d10b6","Type":"ContainerStarted","Data":"bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473"}
Nov 03 11:10:42.996887 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.996721    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd" event={"ID":"c9038b83-5c62-4a08-9a21-1333829d10b6","Type":"ContainerStarted","Data":"77d1106987eaf3a2dba42d71a465795b83d639aa72d66685ec6bec7cdd2c0a6b"}
Nov 03 11:10:42.996887 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.996782    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:42.998395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:42.998368    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m" event={"ID":"bc05d406-c2f5-489f-81e9-c9b32339b6e8","Type":"ContainerStarted","Data":"c9f7e6b6ccd45cd47a1bf273f5fc307db55196fa3c6a1b75c0e24d25f6d7420f"}
Nov 03 11:10:43.014624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:43.014576    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd" podStartSLOduration=3.014557349 podStartE2EDuration="3.014557349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:43.013639751 +0000 UTC m=+2486.918494193" watchObservedRunningTime="2026-11-03 11:10:43.014557349 +0000 UTC m=+2486.919411793"
Nov 03 11:10:43.258542 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:43.258452    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:10:44.009243 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:44.009201    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m" event={"ID":"bc05d406-c2f5-489f-81e9-c9b32339b6e8","Type":"ContainerStarted","Data":"a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db"}
Nov 03 11:10:44.009773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:44.009296    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:44.032349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:44.032295    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m" podStartSLOduration=4.032278408 podStartE2EDuration="4.032278408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:10:44.028239089 +0000 UTC m=+2487.933093556" watchObservedRunningTime="2026-11-03 11:10:44.032278408 +0000 UTC m=+2487.937132849"
Nov 03 11:10:44.490808 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:44.490760    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:46.229965 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:46.229897    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:48.712243 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:48.711077    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"]
Nov 03 11:10:53.711950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:53.711890    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf" podUID="ef711413-5409-4833-a93f-27a2cd8849e3" containerName="payload-processing" containerID="cri-o://e851a9e473c1a0235b7b0b541816e5aa63871efe22f251b8a34c02ef255b9c49" gracePeriod=25
Nov 03 11:10:54.012324 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.012296    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:10:54.100978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.100945    5393 generic.go:356] "Generic (PLEG): container finished" podID="ef711413-5409-4833-a93f-27a2cd8849e3" containerID="e851a9e473c1a0235b7b0b541816e5aa63871efe22f251b8a34c02ef255b9c49" exitCode=0
Nov 03 11:10:54.101176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.101047    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf" event={"ID":"ef711413-5409-4833-a93f-27a2cd8849e3","Type":"ContainerDied","Data":"e851a9e473c1a0235b7b0b541816e5aa63871efe22f251b8a34c02ef255b9c49"}
Nov 03 11:10:54.101176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.101114    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf" event={"ID":"ef711413-5409-4833-a93f-27a2cd8849e3","Type":"ContainerDied","Data":"5f60c324a09bc4eb37086e12fa5182ef42df88e4747bf9db0a893945b67531f4"}
Nov 03 11:10:54.101176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.101136    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f60c324a09bc4eb37086e12fa5182ef42df88e4747bf9db0a893945b67531f4"
Nov 03 11:10:54.105548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.105512    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:10:54.192162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.192119    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-plugins-config-volume\") pod \"ef711413-5409-4833-a93f-27a2cd8849e3\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") "
Nov 03 11:10:54.192363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.192317    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-kube-api-access-bm97s\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-kube-api-access-bm97s\") pod \"ef711413-5409-4833-a93f-27a2cd8849e3\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") "
Nov 03 11:10:54.192530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.192514    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-routing-overlay-volume\") pod \"ef711413-5409-4833-a93f-27a2cd8849e3\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") "
Nov 03 11:10:54.192608 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.192561    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-provider-credentials\") pod \"ef711413-5409-4833-a93f-27a2cd8849e3\" (UID: \"ef711413-5409-4833-a93f-27a2cd8849e3\") "
Nov 03 11:10:54.192757 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.192553    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-plugins-config-volume" pod "ef711413-5409-4833-a93f-27a2cd8849e3" (UID: "ef711413-5409-4833-a93f-27a2cd8849e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:54.192950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.192897    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-routing-overlay-volume" pod "ef711413-5409-4833-a93f-27a2cd8849e3" (UID: "ef711413-5409-4833-a93f-27a2cd8849e3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:10:54.193334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.193314    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:54.193334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.193335    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef711413-5409-4833-a93f-27a2cd8849e3-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:54.194699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.194678    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-kube-api-access-bm97s" pod "ef711413-5409-4833-a93f-27a2cd8849e3" (UID: "ef711413-5409-4833-a93f-27a2cd8849e3"). InnerVolumeSpecName "kube-api-access-bm97s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:54.194892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.194869    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-provider-credentials" pod "ef711413-5409-4833-a93f-27a2cd8849e3" (UID: "ef711413-5409-4833-a93f-27a2cd8849e3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:10:54.295670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.295631    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm97s\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-kube-api-access-bm97s\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:54.295670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:54.295666    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef711413-5409-4833-a93f-27a2cd8849e3-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:10:55.068484 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:55.068450    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:10:55.199946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:55.199440    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"
Nov 03 11:10:55.560007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:55.559124    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"]
Nov 03 11:10:55.583059 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:55.582987    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b76f7948b-5t8cf"]
Nov 03 11:10:55.846000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:55.817186    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"]
Nov 03 11:10:55.924944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:55.924586    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:55.957956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:55.950225    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:55.977942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:55.965645    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.087668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:56.085982    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.128941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:56.128243    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.170959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:56.167687    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 03 11:10:56.529000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:56.528210    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef711413-5409-4833-a93f-27a2cd8849e3" path="/var/lib/kubelet/pods/ef711413-5409-4833-a93f-27a2cd8849e3/volumes"
Nov 03 11:10:56.744984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:56.739053    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.763071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:56.763019    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.973944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:56.950775    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:56.973944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:56.951314    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 11:10:57.022114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.021040    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 11:10:57.155960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.089502    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:57.155960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.131235    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:10:57.185607 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.181839    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Nov 03 11:10:57.185607 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.182329    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 11:10:57.342000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.332515    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:57.439236 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.439203    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:10:57.591264 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:57.590897    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:57.718779 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.718749    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 11:10:57.741188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.739644    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:57.741188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.740260    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 11:10:57.800203 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.799831    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 11:10:57.882069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.882035    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 11:10:57.933942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.933370    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:57.941948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:57.934437    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:10:59.410780 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:10:59.410510    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:10:59.764130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:59.764018    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:10:59.764407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:10:59.764368    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:11:00.131942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:00.131079    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:11:00.131942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:00.131168    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:11:00.488164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:00.488048    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:11:00.860398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:00.860364    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd" podUID="c9038b83-5c62-4a08-9a21-1333829d10b6" containerName="payload-processing" containerID="cri-o://bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473" gracePeriod=25
Nov 03 11:11:01.207692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.207666    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:11:01.325045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.325010    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-plugins-config-volume\") pod \"c9038b83-5c62-4a08-9a21-1333829d10b6\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") "
Nov 03 11:11:01.325200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.325112    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-kube-api-access-b76wf\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-kube-api-access-b76wf\") pod \"c9038b83-5c62-4a08-9a21-1333829d10b6\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") "
Nov 03 11:11:01.325391 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.325370    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-routing-overlay-volume\") pod \"c9038b83-5c62-4a08-9a21-1333829d10b6\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") "
Nov 03 11:11:01.325429 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.325381    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-plugins-config-volume" pod "c9038b83-5c62-4a08-9a21-1333829d10b6" (UID: "c9038b83-5c62-4a08-9a21-1333829d10b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:01.325514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.325492    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-provider-credentials\") pod \"c9038b83-5c62-4a08-9a21-1333829d10b6\" (UID: \"c9038b83-5c62-4a08-9a21-1333829d10b6\") "
Nov 03 11:11:01.325720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.325704    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-routing-overlay-volume" pod "c9038b83-5c62-4a08-9a21-1333829d10b6" (UID: "c9038b83-5c62-4a08-9a21-1333829d10b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:01.326310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.326292    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:01.326401 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.326313    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9038b83-5c62-4a08-9a21-1333829d10b6-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:01.328379 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.328358    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-provider-credentials" pod "c9038b83-5c62-4a08-9a21-1333829d10b6" (UID: "c9038b83-5c62-4a08-9a21-1333829d10b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:01.328460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.328385    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-kube-api-access-b76wf" pod "c9038b83-5c62-4a08-9a21-1333829d10b6" (UID: "c9038b83-5c62-4a08-9a21-1333829d10b6"). InnerVolumeSpecName "kube-api-access-b76wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:01.428418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.428332    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b76wf\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-kube-api-access-b76wf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:01.428418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.428365    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9038b83-5c62-4a08-9a21-1333829d10b6-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:01.788706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.788611    5393 generic.go:356] "Generic (PLEG): container finished" podID="c9038b83-5c62-4a08-9a21-1333829d10b6" containerID="bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473" exitCode=0
Nov 03 11:11:01.788706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.788655    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"
Nov 03 11:11:01.788706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.788686    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd" event={"ID":"c9038b83-5c62-4a08-9a21-1333829d10b6","Type":"ContainerDied","Data":"bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473"}
Nov 03 11:11:01.789350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.788745    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd" event={"ID":"c9038b83-5c62-4a08-9a21-1333829d10b6","Type":"ContainerDied","Data":"77d1106987eaf3a2dba42d71a465795b83d639aa72d66685ec6bec7cdd2c0a6b"}
Nov 03 11:11:01.789350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.788769    5393 scope.go:122] "RemoveContainer" containerID="bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473"
Nov 03 11:11:01.801282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.801260    5393 scope.go:122] "RemoveContainer" containerID="bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473"
Nov 03 11:11:01.802184 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:11:01.802156    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473\": container with ID starting with bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473 not found: ID does not exist" containerID="bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473"
Nov 03 11:11:01.802288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.802195    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473"} err="failed to get container status \"bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473\": rpc error: code = NotFound desc = could not find container \"bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473\": container with ID starting with bcc02be1c9d6106dc5a652fbde54bcb20ec86716a0e164052739466b3333b473 not found: ID does not exist"
Nov 03 11:11:01.817375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.817344    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"]
Nov 03 11:11:01.821518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:01.821494    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458wbjqd"]
Nov 03 11:11:02.233752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:02.233711    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9038b83-5c62-4a08-9a21-1333829d10b6" path="/var/lib/kubelet/pods/c9038b83-5c62-4a08-9a21-1333829d10b6/volumes"
Nov 03 11:11:07.701324 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:11:07.701274    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:11:13.190773 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:11:13.190730    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:11:14.493160 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:11:14.493119    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice/crio-b27d6749da699821d97a20d65bf6fd541938b953ba87d810d181822d23b30cc5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d109b_d30e_49e4_aed0_59eb676c5830.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6777a1e_42f5_4685_bc68_09ae320701b4.slice/crio-conmon-83987dee0ee4fffb62db07f6065d73f7d34544a20dad8ea49b57613e62025c06.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:11:19.277661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:19.277616    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"]
Nov 03 11:11:19.278156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:19.277835    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy" containerID="cri-o://f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4" gracePeriod=30
Nov 03 11:11:24.640994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.634841    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:11:24.806473 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.806436    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-socket\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.806719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.806500    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-certs\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.806719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.806583    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-istio-token\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-istio-token\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.806719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.806644    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e3de284-0506-4d39-9a72-d5da91a31079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3de284-0506-4d39-9a72-d5da91a31079-istio-podinfo\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.806939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.806769    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-socket" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:11:24.806939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.806789    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-data\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.806939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.806833    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-kube-api-access-8mspj\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-kube-api-access-8mspj\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.807132 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.806950    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-certs" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:11:24.807132 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.807045    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3de284-0506-4d39-9a72-d5da91a31079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3de284-0506-4d39-9a72-d5da91a31079-istiod-ca-cert\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.807243 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.807143    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-credential-socket\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.807301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.807281    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-data" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:11:24.807375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.807336    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-envoy\") pod \"9e3de284-0506-4d39-9a72-d5da91a31079\" (UID: \"9e3de284-0506-4d39-9a72-d5da91a31079\") "
Nov 03 11:11:24.807459 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.807419    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-credential-socket" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:11:24.807639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.807592    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3de284-0506-4d39-9a72-d5da91a31079-istiod-ca-cert" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:11:24.808305 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.808282    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3de284-0506-4d39-9a72-d5da91a31079-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:24.808407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.808309    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:24.808407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.808323    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:24.808407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.808337    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:24.808407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.808350    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:24.809553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.809525    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3de284-0506-4d39-9a72-d5da91a31079-istio-podinfo" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:11:24.809682 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.809663    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-istio-token" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:24.810130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.810112    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-kube-api-access-8mspj" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "kube-api-access-8mspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:11:24.810449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.810428    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-envoy" pod "9e3de284-0506-4d39-9a72-d5da91a31079" (UID: "9e3de284-0506-4d39-9a72-d5da91a31079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:11:24.909719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.909685    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3de284-0506-4d39-9a72-d5da91a31079-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:24.909719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.909714    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:24.909719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.909724    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3de284-0506-4d39-9a72-d5da91a31079-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:24.909999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:24.909733    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mspj\" (UniqueName: \"kubernetes.io/projected/9e3de284-0506-4d39-9a72-d5da91a31079-kube-api-access-8mspj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:11:25.107226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.107192    5393 generic.go:356] "Generic (PLEG): container finished" podID="9e3de284-0506-4d39-9a72-d5da91a31079" containerID="f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4" exitCode=0
Nov 03 11:11:25.107399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.107243    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"
Nov 03 11:11:25.107399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.107268    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" event={"ID":"9e3de284-0506-4d39-9a72-d5da91a31079","Type":"ContainerDied","Data":"f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4"}
Nov 03 11:11:25.107399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.107320    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6" event={"ID":"9e3de284-0506-4d39-9a72-d5da91a31079","Type":"ContainerDied","Data":"3e00eb1b6f3b3927cd5d8a79340900063d7a3baef81138716345cbdf93feb80a"}
Nov 03 11:11:25.107399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.107347    5393 scope.go:122] "RemoveContainer" containerID="f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4"
Nov 03 11:11:25.120470 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.120445    5393 scope.go:122] "RemoveContainer" containerID="f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4"
Nov 03 11:11:25.120896 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:11:25.120873    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4\": container with ID starting with f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4 not found: ID does not exist" containerID="f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4"
Nov 03 11:11:25.121015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.120906    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4"} err="failed to get container status \"f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4\": rpc error: code = NotFound desc = could not find container \"f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4\": container with ID starting with f983e79f8b84b7443c912aed08a7efa9603d40b4f1b4167aa6214918044a69d4 not found: ID does not exist"
Nov 03 11:11:25.133143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.133113    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"]
Nov 03 11:11:25.138492 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:25.138461    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-h8xv6"]
Nov 03 11:11:26.234330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:26.234292    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" path="/var/lib/kubelet/pods/9e3de284-0506-4d39-9a72-d5da91a31079/volumes"
Nov 03 11:11:31.731946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:31.720010    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 11:11:31.890954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:31.890041    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:11:31.945400 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:31.943112    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:11:31.945400 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:31.943071    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:11:31.965942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:31.965035    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 11:11:31.987156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:31.985032    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:11:32.090968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:32.090861    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"]
Nov 03 11:11:32.107748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:32.107699    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-762fcd84-fszmh"]
Nov 03 11:11:32.330206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:32.330101    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:11:32.511888 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:32.511343    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacd3c26-3ea0-4a14-b360-deefe220755e" path="/var/lib/kubelet/pods/bacd3c26-3ea0-4a14-b360-deefe220755e/volumes"
Nov 03 11:11:36.309669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.309628    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"]
Nov 03 11:11:36.310667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310643    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c9038b83-5c62-4a08-9a21-1333829d10b6" containerName="payload-processing"
Nov 03 11:11:36.310791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310668    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9038b83-5c62-4a08-9a21-1333829d10b6" containerName="payload-processing"
Nov 03 11:11:36.310791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310726    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ef711413-5409-4833-a93f-27a2cd8849e3" containerName="payload-processing"
Nov 03 11:11:36.310791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310735    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef711413-5409-4833-a93f-27a2cd8849e3" containerName="payload-processing"
Nov 03 11:11:36.310791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310748    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy"
Nov 03 11:11:36.310791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310756    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy"
Nov 03 11:11:36.311082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310959    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9038b83-5c62-4a08-9a21-1333829d10b6" containerName="payload-processing"
Nov 03 11:11:36.311082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310984    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef711413-5409-4833-a93f-27a2cd8849e3" containerName="payload-processing"
Nov 03 11:11:36.311082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.310996    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3de284-0506-4d39-9a72-d5da91a31079" containerName="istio-proxy"
Nov 03 11:11:36.316307 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.316285    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:36.322001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.321975    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"]
Nov 03 11:11:36.322233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.322203    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-2c5853\"/\"e2e-usage-sim-cba7ed-cert\""
Nov 03 11:11:36.322304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.322235    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-2c5853\"/\"kube-root-ca.crt\""
Nov 03 11:11:36.322304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.322278    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-2c5853\"/\"openshift-service-ca.crt\""
Nov 03 11:11:36.322536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.322519    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-2c5853\"/\"default-dockercfg-m4fxb\""
Nov 03 11:11:36.393106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.393068    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ww9\" (UniqueName: \"kubernetes.io/projected/00388998-35db-47ae-91c5-0ddae7e97224-kube-api-access-k8ww9\") pod \"e2e-usage-sim-cba7ed-59647847cb-dzwgq\" (UID: \"00388998-35db-47ae-91c5-0ddae7e97224\") " pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:36.393331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.393173    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/00388998-35db-47ae-91c5-0ddae7e97224-tls\") pod \"e2e-usage-sim-cba7ed-59647847cb-dzwgq\" (UID: \"00388998-35db-47ae-91c5-0ddae7e97224\") " pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:36.494403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.494368    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ww9\" (UniqueName: \"kubernetes.io/projected/00388998-35db-47ae-91c5-0ddae7e97224-kube-api-access-k8ww9\") pod \"e2e-usage-sim-cba7ed-59647847cb-dzwgq\" (UID: \"00388998-35db-47ae-91c5-0ddae7e97224\") " pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:36.494608 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.494419    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/00388998-35db-47ae-91c5-0ddae7e97224-tls\") pod \"e2e-usage-sim-cba7ed-59647847cb-dzwgq\" (UID: \"00388998-35db-47ae-91c5-0ddae7e97224\") " pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:36.497131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.497102    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/00388998-35db-47ae-91c5-0ddae7e97224-tls\") pod \"e2e-usage-sim-cba7ed-59647847cb-dzwgq\" (UID: \"00388998-35db-47ae-91c5-0ddae7e97224\") " pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:36.502992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.502960    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ww9\" (UniqueName: \"kubernetes.io/projected/00388998-35db-47ae-91c5-0ddae7e97224-kube-api-access-k8ww9\") pod \"e2e-usage-sim-cba7ed-59647847cb-dzwgq\" (UID: \"00388998-35db-47ae-91c5-0ddae7e97224\") " pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:36.631739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.631708    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:36.995132 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:11:36.995102    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00388998_35db_47ae_91c5_0ddae7e97224.slice/crio-e7529d76557da992c6116f1ef0e2f5fdb16b6f236e5d2e5bd2127e1fe23b4318 WatchSource:0}: Error finding container e7529d76557da992c6116f1ef0e2f5fdb16b6f236e5d2e5bd2127e1fe23b4318: Status 404 returned error can't find the container with id e7529d76557da992c6116f1ef0e2f5fdb16b6f236e5d2e5bd2127e1fe23b4318
Nov 03 11:11:36.996259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:36.996231    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"]
Nov 03 11:11:37.534776 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:37.534739    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq" event={"ID":"00388998-35db-47ae-91c5-0ddae7e97224","Type":"ContainerStarted","Data":"1e30d4171a2fb87f26bd6ba4fa150fa483fd3cfcca39aba3bac24eaa8e3859a2"}
Nov 03 11:11:37.535407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:37.534786    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq" event={"ID":"00388998-35db-47ae-91c5-0ddae7e97224","Type":"ContainerStarted","Data":"e7529d76557da992c6116f1ef0e2f5fdb16b6f236e5d2e5bd2127e1fe23b4318"}
Nov 03 11:11:37.535407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:37.534816    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:37.540099 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:37.540075    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:11:37.549908 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:37.549859    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq" podStartSLOduration=1.549840055 podStartE2EDuration="1.549840055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:11:37.548783876 +0000 UTC m=+2541.453638317" watchObservedRunningTime="2026-11-03 11:11:37.549840055 +0000 UTC m=+2541.454694496"
Nov 03 11:11:43.113823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.113783    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"]
Nov 03 11:11:43.120014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.119983    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.121876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.121854    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"]
Nov 03 11:11:43.144315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.144285    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cbeebd\""
Nov 03 11:11:43.145048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.144614    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"routing-overlay\""
Nov 03 11:11:43.145201 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.144668    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"openshift-service-ca.crt\""
Nov 03 11:11:43.145301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.144839    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"payload-processing-external-model-e2e--8c972630-dockercfg-mt5gm\""
Nov 03 11:11:43.145367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.144906    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"kube-root-ca.crt\""
Nov 03 11:11:43.145367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.145003    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cbeebd\"/\"agc-cred-9b86094f8fce\""
Nov 03 11:11:43.303848 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.303809    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.331196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.331164    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.331640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.331603    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.331858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.331845    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47gfm\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-kube-api-access-47gfm\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.523564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.516483    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.523564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.516703    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.523564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.516867    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.523564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.516991    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47gfm\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-kube-api-access-47gfm\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.523564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.518236    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.523564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.523198    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.572949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.572897    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.626110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.622892    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47gfm\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-kube-api-access-47gfm\") pod \"payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") " pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.745666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:43.745631    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:43.889655 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:11:43.889618    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:11:44.322698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:44.322651    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"]
Nov 03 11:11:44.324133 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:11:44.324103    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2f2b77_7d92_47c1_aa7d_da5596769d13.slice/crio-b7c5661bb80899ac0db5ca8053796601ccd7f81ce4e81ce9eaa951e46164e5c8 WatchSource:0}: Error finding container b7c5661bb80899ac0db5ca8053796601ccd7f81ce4e81ce9eaa951e46164e5c8: Status 404 returned error can't find the container with id b7c5661bb80899ac0db5ca8053796601ccd7f81ce4e81ce9eaa951e46164e5c8
Nov 03 11:11:44.712086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:44.712033    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w" event={"ID":"5a2f2b77-7d92-47c1-aa7d-da5596769d13","Type":"ContainerStarted","Data":"64ad7b5e54a5335c7d26e3426c8318800b0453f208ef16c1f3e1dad2197990ce"}
Nov 03 11:11:44.712272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:44.712117    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:11:44.712272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:44.712142    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w" event={"ID":"5a2f2b77-7d92-47c1-aa7d-da5596769d13","Type":"ContainerStarted","Data":"b7c5661bb80899ac0db5ca8053796601ccd7f81ce4e81ce9eaa951e46164e5c8"}
Nov 03 11:11:44.735105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:44.735053    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w" podStartSLOduration=1.735041275 podStartE2EDuration="1.735041275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:11:44.731549456 +0000 UTC m=+2548.636403910" watchObservedRunningTime="2026-11-03 11:11:44.735041275 +0000 UTC m=+2548.639895751"
Nov 03 11:11:55.735943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:11:55.733718    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:12:02.597641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:02.597553    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"]
Nov 03 11:12:02.598167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:02.597783    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy" containerID="cri-o://6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc" gracePeriod=30
Nov 03 11:12:08.101912 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.101882    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 11:12:08.199556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.199519    5393 generic.go:356] "Generic (PLEG): container finished" podID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerID="6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc" exitCode=0
Nov 03 11:12:08.199778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.199579    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" event={"ID":"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a","Type":"ContainerDied","Data":"6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc"}
Nov 03 11:12:08.199778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.199651    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn" event={"ID":"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a","Type":"ContainerDied","Data":"d405b1dbcdf25b191659c1c3acdc6245761fbbe3041a76084f73acd36fd25788"}
Nov 03 11:12:08.199778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.199693    5393 scope.go:122] "RemoveContainer" containerID="6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc"
Nov 03 11:12:08.200048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.199874    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"
Nov 03 11:12:08.217133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.217104    5393 scope.go:122] "RemoveContainer" containerID="6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc"
Nov 03 11:12:08.217463 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:12:08.217437    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc\": container with ID starting with 6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc not found: ID does not exist" containerID="6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc"
Nov 03 11:12:08.217569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.217475    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc"} err="failed to get container status \"6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc\": rpc error: code = NotFound desc = could not find container \"6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc\": container with ID starting with 6aaed04cee4ab2b99264fb06d41c200989cb3ad7a33219fe15b51fee7b051ccc not found: ID does not exist"
Nov 03 11:12:08.233348 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233283    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-kube-api-access-sdw47\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-kube-api-access-sdw47\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.233479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233356    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-certs\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.233479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233402    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-token\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.233479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233447    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-envoy\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.233689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233503    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-credential-socket\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.233689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233551    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-socket\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.233812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233719    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-podinfo\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.233812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233761    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istiod-ca-cert\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.233976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.233812    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-data\") pod \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\" (UID: \"2ae62e87-8603-4fc9-8c0a-0a751df1ce7a\") "
Nov 03 11:12:08.234163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.234136    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-certs" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:08.234319 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.234302    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-data" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:08.234529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.234508    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-socket" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:08.234752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.234731    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.234812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.234759    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.234812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.234772    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.235253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.235231    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-credential-socket" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:08.235725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.235702    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istiod-ca-cert" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:12:08.236444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.236396    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-envoy" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:12:08.237288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.237256    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-kube-api-access-sdw47" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "kube-api-access-sdw47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:08.237388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.237292    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-token" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:12:08.237464 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.237391    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-podinfo" pod "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" (UID: "2ae62e87-8603-4fc9-8c0a-0a751df1ce7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:12:08.337229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.337187    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.337434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.337334    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.337434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.337358    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.337434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.337370    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdw47\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-kube-api-access-sdw47\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.337434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.337384    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.337434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.337400    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:12:08.532592 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.532557    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"]
Nov 03 11:12:08.539182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:08.539155    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-c46kn"]
Nov 03 11:12:10.234052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:10.234019    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" path="/var/lib/kubelet/pods/2ae62e87-8603-4fc9-8c0a-0a751df1ce7a/volumes"
Nov 03 11:12:13.220406 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:12:13.220367    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:12:22.537931 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.537878    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"]
Nov 03 11:12:22.538705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.538689    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy"
Nov 03 11:12:22.538751 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.538706    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy"
Nov 03 11:12:22.538833 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.538824    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae62e87-8603-4fc9-8c0a-0a751df1ce7a" containerName="istio-proxy"
Nov 03 11:12:22.544363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.544330    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:22.549482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.549427    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"]
Nov 03 11:12:22.549640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.549569    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9265bf\""
Nov 03 11:12:22.549640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.549612    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9265bf-dockercfg-bs9cw\""
Nov 03 11:12:22.561681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.561647    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"]
Nov 03 11:12:22.568227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.568197    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:22.579639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.579607    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"]
Nov 03 11:12:22.827139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.827039    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtk6g\" (UniqueName: \"kubernetes.io/projected/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-kube-api-access-dtk6g\") pod \"payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms\" (UID: \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:22.834292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.834238    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms\" (UID: \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:22.834701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.834677    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237bdc8b-3f3f-4926-93bc-a23b09b58829-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq\" (UID: \"237bdc8b-3f3f-4926-93bc-a23b09b58829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:22.834947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.834910    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlx6\" (UniqueName: \"kubernetes.io/projected/237bdc8b-3f3f-4926-93bc-a23b09b58829-kube-api-access-vdlx6\") pod \"payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq\" (UID: \"237bdc8b-3f3f-4926-93bc-a23b09b58829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:22.952758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.952210    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtk6g\" (UniqueName: \"kubernetes.io/projected/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-kube-api-access-dtk6g\") pod \"payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms\" (UID: \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:22.952758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.952390    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms\" (UID: \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:22.952758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.952487    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237bdc8b-3f3f-4926-93bc-a23b09b58829-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq\" (UID: \"237bdc8b-3f3f-4926-93bc-a23b09b58829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:22.952758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.952523    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlx6\" (UniqueName: \"kubernetes.io/projected/237bdc8b-3f3f-4926-93bc-a23b09b58829-kube-api-access-vdlx6\") pod \"payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq\" (UID: \"237bdc8b-3f3f-4926-93bc-a23b09b58829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:22.954327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.954267    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms\" (UID: \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:22.954327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.954274    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237bdc8b-3f3f-4926-93bc-a23b09b58829-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq\" (UID: \"237bdc8b-3f3f-4926-93bc-a23b09b58829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:22.966543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.966518    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlx6\" (UniqueName: \"kubernetes.io/projected/237bdc8b-3f3f-4926-93bc-a23b09b58829-kube-api-access-vdlx6\") pod \"payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq\" (UID: \"237bdc8b-3f3f-4926-93bc-a23b09b58829\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:22.966673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:22.966608    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtk6g\" (UniqueName: \"kubernetes.io/projected/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-kube-api-access-dtk6g\") pod \"payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms\" (UID: \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:23.162865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:23.162822    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:23.196669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:23.196629    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:23.466303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:23.466276    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"]
Nov 03 11:12:23.466528 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:12:23.466499    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237bdc8b_3f3f_4926_93bc_a23b09b58829.slice/crio-1df04334c3e4c5e6dce9449ac611b25866c64dd7b1bc5124c78235ae317dc3bf WatchSource:0}: Error finding container 1df04334c3e4c5e6dce9449ac611b25866c64dd7b1bc5124c78235ae317dc3bf: Status 404 returned error can't find the container with id 1df04334c3e4c5e6dce9449ac611b25866c64dd7b1bc5124c78235ae317dc3bf
Nov 03 11:12:23.514279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:23.514253    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"]
Nov 03 11:12:23.522512 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:12:23.522478    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b6d65f_4cb1_4ce5_8321_a3a4b812e22d.slice/crio-b916630c349b138508e7cfc1aecca8ff633a0cd4fff7002d76814ea1b78429b0 WatchSource:0}: Error finding container b916630c349b138508e7cfc1aecca8ff633a0cd4fff7002d76814ea1b78429b0: Status 404 returned error can't find the container with id b916630c349b138508e7cfc1aecca8ff633a0cd4fff7002d76814ea1b78429b0
Nov 03 11:12:24.341966 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:24.341903    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq" event={"ID":"237bdc8b-3f3f-4926-93bc-a23b09b58829","Type":"ContainerStarted","Data":"aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879"}
Nov 03 11:12:24.342432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:24.341989    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq" event={"ID":"237bdc8b-3f3f-4926-93bc-a23b09b58829","Type":"ContainerStarted","Data":"1df04334c3e4c5e6dce9449ac611b25866c64dd7b1bc5124c78235ae317dc3bf"}
Nov 03 11:12:24.342432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:24.342017    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:24.343695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:24.343661    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms" event={"ID":"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d","Type":"ContainerStarted","Data":"301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac"}
Nov 03 11:12:24.343833 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:24.343717    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms" event={"ID":"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d","Type":"ContainerStarted","Data":"b916630c349b138508e7cfc1aecca8ff633a0cd4fff7002d76814ea1b78429b0"}
Nov 03 11:12:24.343833 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:24.343738    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:24.358327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:24.358264    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq" podStartSLOduration=2.358234294 podStartE2EDuration="2.358234294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:12:24.357364996 +0000 UTC m=+2588.262219461" watchObservedRunningTime="2026-11-03 11:12:24.358234294 +0000 UTC m=+2588.263088737"
Nov 03 11:12:24.373588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:24.373528    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms" podStartSLOduration=2.373510614 podStartE2EDuration="2.373510614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:12:24.372141812 +0000 UTC m=+2588.276996246" watchObservedRunningTime="2026-11-03 11:12:24.373510614 +0000 UTC m=+2588.278365056"
Nov 03 11:12:30.116242 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:30.116068    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-hs9mc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:12:30.116242 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:30.116158    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-hs9mc" podUID="7afc9a85-df4f-4401-8656-19750d21605a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:12:35.356632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:35.356596    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:12:35.357104 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:12:35.356670    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:12:43.246336 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:12:43.245057    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:13:12.710039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.710003    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"]
Nov 03 11:13:12.721415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.721379    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:12.731214 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.728028    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-60ad03-dockercfg-cw7vn\""
Nov 03 11:13:12.743530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.743069    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"]
Nov 03 11:13:12.769840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.769801    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hpv\" (UniqueName: \"kubernetes.io/projected/1a422450-f210-44dc-9226-43532342e514-kube-api-access-t4hpv\") pod \"payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w\" (UID: \"1a422450-f210-44dc-9226-43532342e514\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:12.770170 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.770108    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a422450-f210-44dc-9226-43532342e514-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w\" (UID: \"1a422450-f210-44dc-9226-43532342e514\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:12.809941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.809007    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-60ad03\""
Nov 03 11:13:12.900941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.900877    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a422450-f210-44dc-9226-43532342e514-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w\" (UID: \"1a422450-f210-44dc-9226-43532342e514\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:12.913188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.902226    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a422450-f210-44dc-9226-43532342e514-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w\" (UID: \"1a422450-f210-44dc-9226-43532342e514\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:12.949591 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:12.949545    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hpv\" (UniqueName: \"kubernetes.io/projected/1a422450-f210-44dc-9226-43532342e514-kube-api-access-t4hpv\") pod \"payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w\" (UID: \"1a422450-f210-44dc-9226-43532342e514\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:13.021995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:13.007773    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hpv\" (UniqueName: \"kubernetes.io/projected/1a422450-f210-44dc-9226-43532342e514-kube-api-access-t4hpv\") pod \"payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w\" (UID: \"1a422450-f210-44dc-9226-43532342e514\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:13.048696 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:13.048671    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:13.197733 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:13.197694    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:13:13.438031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:13.438000    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"]
Nov 03 11:13:13.438424 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:13:13.438400    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a422450_f210_44dc_9226_43532342e514.slice/crio-985e02b1bf0b8163dcbc0b334a9dcdac1bab1ba7453a9b86088815edbd301c20 WatchSource:0}: Error finding container 985e02b1bf0b8163dcbc0b334a9dcdac1bab1ba7453a9b86088815edbd301c20: Status 404 returned error can't find the container with id 985e02b1bf0b8163dcbc0b334a9dcdac1bab1ba7453a9b86088815edbd301c20
Nov 03 11:13:14.262807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:14.262752    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w" event={"ID":"1a422450-f210-44dc-9226-43532342e514","Type":"ContainerStarted","Data":"14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047"}
Nov 03 11:13:14.263357 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:14.262840    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w" event={"ID":"1a422450-f210-44dc-9226-43532342e514","Type":"ContainerStarted","Data":"985e02b1bf0b8163dcbc0b334a9dcdac1bab1ba7453a9b86088815edbd301c20"}
Nov 03 11:13:14.263357 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:14.263026    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:14.281778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:14.281706    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w" podStartSLOduration=2.281688109 podStartE2EDuration="2.281688109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:14.279753836 +0000 UTC m=+2638.184608280" watchObservedRunningTime="2026-11-03 11:13:14.281688109 +0000 UTC m=+2638.186542554"
Nov 03 11:13:20.212175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:20.212132    5393 scope.go:122] "RemoveContainer" containerID="da8a3f24ae87d561465f76cf3b788647445f147a4f78b15cd4c3d059b1323a83"
Nov 03 11:13:25.274754 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:25.274720    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:13:29.455395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.455204    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"]
Nov 03 11:13:29.469266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.469237    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.470680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.470657    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"]
Nov 03 11:13:29.476984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.476961    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-60ad03-serving-cert\""
Nov 03 11:13:29.602718 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.602682    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-metrics-certs\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.602962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.602842    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b8a2f-1255-4bf0-afca-616548a97736-openshift-service-ca\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.603140 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.603118    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9d5\" (UniqueName: \"kubernetes.io/projected/ff6b8a2f-1255-4bf0-afca-616548a97736-kube-api-access-fv9d5\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.603185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.603170    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-maas-api-tls\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.738829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.738721    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-maas-api-tls\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.739072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.738889    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-metrics-certs\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.739072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.739013    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b8a2f-1255-4bf0-afca-616548a97736-openshift-service-ca\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.739329 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.739294    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9d5\" (UniqueName: \"kubernetes.io/projected/ff6b8a2f-1255-4bf0-afca-616548a97736-kube-api-access-fv9d5\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.753941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.740785    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b8a2f-1255-4bf0-afca-616548a97736-openshift-service-ca\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.754470 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.754440    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-maas-api-tls\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.761315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.761283    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9d5\" (UniqueName: \"kubernetes.io/projected/ff6b8a2f-1255-4bf0-afca-616548a97736-kube-api-access-fv9d5\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.764358 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.764329    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-metrics-certs\") pod \"maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:29.794499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:29.794473    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:30.200717 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:30.200683    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"]
Nov 03 11:13:30.206336 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:13:30.206307    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6b8a2f_1255_4bf0_afca_616548a97736.slice/crio-ea95d8278176bf4a4420de56f7a757037fd538e7fdcb9d97e0b5ddbc0b328e43 WatchSource:0}: Error finding container ea95d8278176bf4a4420de56f7a757037fd538e7fdcb9d97e0b5ddbc0b328e43: Status 404 returned error can't find the container with id ea95d8278176bf4a4420de56f7a757037fd538e7fdcb9d97e0b5ddbc0b328e43
Nov 03 11:13:30.208446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:30.208427    5393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:13:30.527813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:30.527044    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz" event={"ID":"ff6b8a2f-1255-4bf0-afca-616548a97736","Type":"ContainerStarted","Data":"ea95d8278176bf4a4420de56f7a757037fd538e7fdcb9d97e0b5ddbc0b328e43"}
Nov 03 11:13:31.537939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:31.537890    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz" event={"ID":"ff6b8a2f-1255-4bf0-afca-616548a97736","Type":"ContainerStarted","Data":"a5f1774ee6c861bf00f25c74f593ce859866105872b175016b47729c32a94d55"}
Nov 03 11:13:31.538392 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:31.537975    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:31.558886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:31.558830    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz" podStartSLOduration=2.10235897 podStartE2EDuration="2.558816983s" totalImagesPullingTime="456.458013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:31.554904295 +0000 UTC m=+2655.459758749" watchObservedRunningTime="2026-11-03 11:13:31.558816983 +0000 UTC m=+2655.463671424"
Nov 03 11:13:37.556594 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:37.556565    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:13:42.064460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.064421    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"]
Nov 03 11:13:42.071500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.071470    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.077234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.072981    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"]
Nov 03 11:13:42.093823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.093791    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"agc-cred-6096dd351669\""
Nov 03 11:13:42.094060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.093853    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"agc-cred-ad4ca937633f\""
Nov 03 11:13:42.094060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.093891    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"agc-cred-b8c3e972a39f\""
Nov 03 11:13:42.094060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.094011    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"payload-processing-external-model-e2e--7ca7b2d5-dockercfg-cvxnx\""
Nov 03 11:13:42.094260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.094104    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5711cf\""
Nov 03 11:13:42.094260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.094199    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"routing-overlay\""
Nov 03 11:13:42.094376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.094287    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"openshift-service-ca.crt\""
Nov 03 11:13:42.094673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.094658    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"kube-root-ca.crt\""
Nov 03 11:13:42.094822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.094769    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.094892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.094844    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvm4\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-kube-api-access-gcvm4\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.095211 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.095184    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.095378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.095313    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.196802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.196765    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.197004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.196872    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.197069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.197041    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.197120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.197069    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvm4\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-kube-api-access-gcvm4\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.197706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.197680    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.197843 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.197757    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.199638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.199614    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.208883 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.208848    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvm4\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-kube-api-access-gcvm4\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.415827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.415785    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:42.704372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:42.704223    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"]
Nov 03 11:13:42.704646 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:13:42.704612    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36865103_5ff2_4b75_8b35_81c900352b4e.slice/crio-a75412441d44fa480cb433b4d053157474de2ffd7ddc029cdfa4850423f5ea6e WatchSource:0}: Error finding container a75412441d44fa480cb433b4d053157474de2ffd7ddc029cdfa4850423f5ea6e: Status 404 returned error can't find the container with id a75412441d44fa480cb433b4d053157474de2ffd7ddc029cdfa4850423f5ea6e
Nov 03 11:13:43.191494 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:43.191453    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:13:43.688172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:43.688132    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" event={"ID":"36865103-5ff2-4b75-8b35-81c900352b4e","Type":"ContainerStarted","Data":"f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12"}
Nov 03 11:13:43.688435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:43.688203    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" event={"ID":"36865103-5ff2-4b75-8b35-81c900352b4e","Type":"ContainerStarted","Data":"a75412441d44fa480cb433b4d053157474de2ffd7ddc029cdfa4850423f5ea6e"}
Nov 03 11:13:43.688435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:43.688257    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:43.707352 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:43.707296    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" podStartSLOduration=1.707280658 podStartE2EDuration="1.707280658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:43.704828412 +0000 UTC m=+2667.609682854" watchObservedRunningTime="2026-11-03 11:13:43.707280658 +0000 UTC m=+2667.612135099"
Nov 03 11:13:43.763333 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:43.763299    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5711cf/agc-cred-ad4ca937633f: secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:43.763333 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:43.763332    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn: secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:43.763616 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:43.763390    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials podName:36865103-5ff2-4b75-8b35-81c900352b4e nodeName:}" failed. No retries permitted until 2026-11-03 11:13:44.263370028 +0000 UTC m=+2668.168224464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" (UID: "36865103-5ff2-4b75-8b35-81c900352b4e") : secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:44.033279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:44.033181    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"]
Nov 03 11:13:44.506939 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:44.505061    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5711cf/agc-cred-ad4ca937633f: secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:44.506939 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:44.505129    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn: secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:44.518021 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:44.515850    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials podName:36865103-5ff2-4b75-8b35-81c900352b4e nodeName:}" failed. No retries permitted until 2026-11-03 11:13:45.515818247 +0000 UTC m=+2669.420672672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" (UID: "36865103-5ff2-4b75-8b35-81c900352b4e") : secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:45.609880 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:45.609840    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5711cf/agc-cred-ad4ca937633f: secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:45.609880 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:45.609874    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn: secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:45.610360 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:45.609975    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials podName:36865103-5ff2-4b75-8b35-81c900352b4e nodeName:}" failed. No retries permitted until 2026-11-03 11:13:47.609960032 +0000 UTC m=+2671.514814463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" (UID: "36865103-5ff2-4b75-8b35-81c900352b4e") : secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:47.657631 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:47.657595    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5711cf/agc-cred-ad4ca937633f: secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:47.657631 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:47.657626    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn: secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:47.658128 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:47.657683    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials podName:36865103-5ff2-4b75-8b35-81c900352b4e nodeName:}" failed. No retries permitted until 2026-11-03 11:13:51.657664676 +0000 UTC m=+2675.562519096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" (UID: "36865103-5ff2-4b75-8b35-81c900352b4e") : secret "agc-cred-ad4ca937633f" not found
Nov 03 11:13:50.706193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:50.706148    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" podUID="36865103-5ff2-4b75-8b35-81c900352b4e" containerName="payload-processing" containerID="cri-o://f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12" gracePeriod=25
Nov 03 11:13:51.250943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.244101    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:51.355115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.355083    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-kube-api-access-gcvm4\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-kube-api-access-gcvm4\") pod \"36865103-5ff2-4b75-8b35-81c900352b4e\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") "
Nov 03 11:13:51.355322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.355148    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-routing-overlay-volume\") pod \"36865103-5ff2-4b75-8b35-81c900352b4e\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") "
Nov 03 11:13:51.355322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.355235    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-plugins-config-volume\") pod \"36865103-5ff2-4b75-8b35-81c900352b4e\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") "
Nov 03 11:13:51.355498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.355479    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials\") pod \"36865103-5ff2-4b75-8b35-81c900352b4e\" (UID: \"36865103-5ff2-4b75-8b35-81c900352b4e\") "
Nov 03 11:13:51.355650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.355545    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-routing-overlay-volume" pod "36865103-5ff2-4b75-8b35-81c900352b4e" (UID: "36865103-5ff2-4b75-8b35-81c900352b4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:13:51.355789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.355769    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-plugins-config-volume" pod "36865103-5ff2-4b75-8b35-81c900352b4e" (UID: "36865103-5ff2-4b75-8b35-81c900352b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:13:51.357118 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.357094    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:51.357118 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.357121    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36865103-5ff2-4b75-8b35-81c900352b4e-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:51.357865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.357845    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-kube-api-access-gcvm4" pod "36865103-5ff2-4b75-8b35-81c900352b4e" (UID: "36865103-5ff2-4b75-8b35-81c900352b4e"). InnerVolumeSpecName "kube-api-access-gcvm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:13:51.358481 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.358463    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials" pod "36865103-5ff2-4b75-8b35-81c900352b4e" (UID: "36865103-5ff2-4b75-8b35-81c900352b4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:13:51.471250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.467965    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:51.471250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.468003    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvm4\" (UniqueName: \"kubernetes.io/projected/36865103-5ff2-4b75-8b35-81c900352b4e-kube-api-access-gcvm4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:13:51.778760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.778672    5393 generic.go:356] "Generic (PLEG): container finished" podID="36865103-5ff2-4b75-8b35-81c900352b4e" containerID="f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12" exitCode=0
Nov 03 11:13:51.778760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.778720    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"
Nov 03 11:13:51.779270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.778750    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" event={"ID":"36865103-5ff2-4b75-8b35-81c900352b4e","Type":"ContainerDied","Data":"f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12"}
Nov 03 11:13:51.779270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.778811    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn" event={"ID":"36865103-5ff2-4b75-8b35-81c900352b4e","Type":"ContainerDied","Data":"a75412441d44fa480cb433b4d053157474de2ffd7ddc029cdfa4850423f5ea6e"}
Nov 03 11:13:51.779270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.778840    5393 scope.go:122] "RemoveContainer" containerID="f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12"
Nov 03 11:13:51.791381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.791361    5393 scope.go:122] "RemoveContainer" containerID="f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12"
Nov 03 11:13:51.791710 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:13:51.791691    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12\": container with ID starting with f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12 not found: ID does not exist" containerID="f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12"
Nov 03 11:13:51.791770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.791719    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12"} err="failed to get container status \"f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12\": rpc error: code = NotFound desc = could not find container \"f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12\": container with ID starting with f2985c5507ecaa7150c91e0ac4a6157cf9c8aa304daea266eb0f352bda5f3a12 not found: ID does not exist"
Nov 03 11:13:51.806399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.806362    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"]
Nov 03 11:13:51.810467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:51.810444    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-676cmgznn"]
Nov 03 11:13:52.254191 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:52.254158    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36865103-5ff2-4b75-8b35-81c900352b4e" path="/var/lib/kubelet/pods/36865103-5ff2-4b75-8b35-81c900352b4e/volumes"
Nov 03 11:13:53.323483 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.322283    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"]
Nov 03 11:13:53.324013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.323546    5393 cpu_manager.go:414] "containerMap: removing container" podUID="36865103-5ff2-4b75-8b35-81c900352b4e" containerName="payload-processing"
Nov 03 11:13:53.324013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.323568    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36865103-5ff2-4b75-8b35-81c900352b4e" containerName="payload-processing"
Nov 03 11:13:53.324013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.323864    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="36865103-5ff2-4b75-8b35-81c900352b4e" containerName="payload-processing"
Nov 03 11:13:53.347985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.347888    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.352346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.351591    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"]
Nov 03 11:13:53.358439 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.358413    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9265bf\"/\"test-model-9265bf-kserve-self-signed-certs\""
Nov 03 11:13:53.369589 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.362113    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9265bf\"/\"kube-root-ca.crt\""
Nov 03 11:13:53.369773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.362744    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9265bf\"/\"openshift-service-ca.crt\""
Nov 03 11:13:53.369773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.362838    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9265bf\"/\"default-dockercfg-wnxjg\""
Nov 03 11:13:53.421857 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.421821    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-tmp-dir\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.421857 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.421854    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a45c61-35ab-4b96-8758-c84a6f6d3450-tls-certs\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.422142 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.421882    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-home\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.422142 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.422129    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-model-cache\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.422274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.422243    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ms2x\" (UniqueName: \"kubernetes.io/projected/60a45c61-35ab-4b96-8758-c84a6f6d3450-kube-api-access-8ms2x\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.422440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.422410    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-dshm\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.528947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.528226    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-dshm\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.528947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.528394    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-tmp-dir\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.528947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.528419    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a45c61-35ab-4b96-8758-c84a6f6d3450-tls-certs\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.528947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.528464    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-home\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.528947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.528579    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-model-cache\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.528947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.528700    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ms2x\" (UniqueName: \"kubernetes.io/projected/60a45c61-35ab-4b96-8758-c84a6f6d3450-kube-api-access-8ms2x\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.556357 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.556320    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-tmp-dir\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.557278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.557253    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-model-cache\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.557664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.557645    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-home\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.615004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.595834    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ms2x\" (UniqueName: \"kubernetes.io/projected/60a45c61-35ab-4b96-8758-c84a6f6d3450-kube-api-access-8ms2x\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.615004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.607235    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-dshm\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.615004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.607745    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a45c61-35ab-4b96-8758-c84a6f6d3450-tls-certs\") pod \"test-model-9265bf-kserve-546f64db67-lc78k\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") " pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:53.685638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:53.685604    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:13:54.027434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:54.027407    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"]
Nov 03 11:13:54.028537 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:13:54.028509    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60a45c61_35ab_4b96_8758_c84a6f6d3450.slice/crio-6088400cdf44e9b3b703c1e307dd1832f2321c51a65a71d69d1c3290e475c13a WatchSource:0}: Error finding container 6088400cdf44e9b3b703c1e307dd1832f2321c51a65a71d69d1c3290e475c13a: Status 404 returned error can't find the container with id 6088400cdf44e9b3b703c1e307dd1832f2321c51a65a71d69d1c3290e475c13a
Nov 03 11:13:54.809874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:54.809831    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k" event={"ID":"60a45c61-35ab-4b96-8758-c84a6f6d3450","Type":"ContainerStarted","Data":"cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc"}
Nov 03 11:13:54.810490 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:54.809888    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k" event={"ID":"60a45c61-35ab-4b96-8758-c84a6f6d3450","Type":"ContainerStarted","Data":"6088400cdf44e9b3b703c1e307dd1832f2321c51a65a71d69d1c3290e475c13a"}
Nov 03 11:13:54.831863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:54.831804    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k" podStartSLOduration=1.831788553 podStartE2EDuration="1.831788553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:13:54.831273673 +0000 UTC m=+2678.736128126" watchObservedRunningTime="2026-11-03 11:13:54.831788553 +0000 UTC m=+2678.736642993"
Nov 03 11:13:55.687042 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:13:55.687003    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:14:03.686810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:03.686773    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:14:03.700753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:03.700724    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:14:03.988285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:03.988194    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:14:05.861824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:05.861786    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"]
Nov 03 11:14:05.873311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:05.870397    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:05.877465 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:05.876579    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-60ad03\"/\"openshift-service-ca.crt\""
Nov 03 11:14:05.877652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:05.877600    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-60ad03\"/\"kube-root-ca.crt\""
Nov 03 11:14:05.877737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:05.877703    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-60ad03\"/\"default-dockercfg-htvbt\""
Nov 03 11:14:05.877810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:05.877788    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-60ad03\"/\"test-model-60ad03-kserve-self-signed-certs\""
Nov 03 11:14:05.889114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:05.879093    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"]
Nov 03 11:14:06.025640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.025597    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-model-cache\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.025981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.025958    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-dshm\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.026151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.026125    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tls-certs\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.026232 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.026216    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-home\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.026292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.026247    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tmp-dir\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.026350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.026291    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7gm\" (UniqueName: \"kubernetes.io/projected/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-kube-api-access-lh7gm\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.128423 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.128297    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7gm\" (UniqueName: \"kubernetes.io/projected/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-kube-api-access-lh7gm\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.128629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.128431    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-model-cache\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.128712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.128648    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-dshm\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.128777 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.128723    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tls-certs\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.128846 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.128820    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-home\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.128904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.128864    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tmp-dir\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.129009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.128979    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-model-cache\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.129293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.129271    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-home\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.129447 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.129309    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tmp-dir\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.131301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.131275    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-dshm\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.131834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.131819    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tls-certs\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.136715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.136693    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7gm\" (UniqueName: \"kubernetes.io/projected/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-kube-api-access-lh7gm\") pod \"test-model-60ad03-kserve-7958b58c99-zvf8p\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") " pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:06.209594 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:06.209283    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:07.008973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.008869    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"]
Nov 03 11:14:07.128674 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.128649    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"]
Nov 03 11:14:07.144872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.138384    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.147684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.146725    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"]
Nov 03 11:14:07.162293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.161380    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"routing-overlay\""
Nov 03 11:14:07.162293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.161468    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"kube-root-ca.crt\""
Nov 03 11:14:07.162293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.161555    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5711cf\""
Nov 03 11:14:07.162293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.161612    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"openshift-service-ca.crt\""
Nov 03 11:14:07.163021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.162992    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"agc-cred-b8c3e972a39f\""
Nov 03 11:14:07.163271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.163218    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5711cf\"/\"payload-processing-external-model-e2e--7ca7b2d5-dockercfg-cvxnx\""
Nov 03 11:14:07.399942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.357093    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.400670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.400634    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxf5\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-kube-api-access-9xxf5\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.426946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.421059    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.426946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.421161    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.541102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.528028    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxf5\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-kube-api-access-9xxf5\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.541102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.528090    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.541102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.528173    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.541102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.528300    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.559313 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.542110    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.640886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.612849    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.717793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.708015    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxf5\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-kube-api-access-9xxf5\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:07.793836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:07.766506    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") " pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:08.014938 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:08.014681    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:08.119111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:08.119073    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p" event={"ID":"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8","Type":"ContainerStarted","Data":"d1987ed1eee83ac2e79c309abda65e0424d6e1bc0bb57e7c1b174c36001e5e33"}
Nov 03 11:14:08.119355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:08.119328    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p" event={"ID":"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8","Type":"ContainerStarted","Data":"636d5a4923999863ec7c9c3cdf7edb58abec4dde172723be8c99aa2095277d33"}
Nov 03 11:14:08.147895 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:08.147820    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p" podStartSLOduration=3.147801734 podStartE2EDuration="3.147801734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:08.141869363 +0000 UTC m=+2692.046723805" watchObservedRunningTime="2026-11-03 11:14:08.147801734 +0000 UTC m=+2692.052656178"
Nov 03 11:14:08.210142 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:08.210105    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:08.424844 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:08.424811    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"]
Nov 03 11:14:08.426363 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:14:08.426328    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice/crio-e863d517002d958fd0bc26eb7be20e43e3420eac01710d5ed5ea2b7112ef38a3 WatchSource:0}: Error finding container e863d517002d958fd0bc26eb7be20e43e3420eac01710d5ed5ea2b7112ef38a3: Status 404 returned error can't find the container with id e863d517002d958fd0bc26eb7be20e43e3420eac01710d5ed5ea2b7112ef38a3
Nov 03 11:14:08.914944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:08.912940    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"]
Nov 03 11:14:09.291574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:09.291446    5393 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" secret="" err="secret \"payload-processing-external-model-e2e--7ca7b2d5-dockercfg-cvxnx\" not found"
Nov 03 11:14:09.357962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:09.308990    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" event={"ID":"7fcfb069-7210-45e2-a830-5b4294e0ab20","Type":"ContainerStarted","Data":"b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14"}
Nov 03 11:14:09.357962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:09.353037    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" event={"ID":"7fcfb069-7210-45e2-a830-5b4294e0ab20","Type":"ContainerStarted","Data":"e863d517002d958fd0bc26eb7be20e43e3420eac01710d5ed5ea2b7112ef38a3"}
Nov 03 11:14:09.357962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:09.353089    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:09.357962 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.330087    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5711cf/agc-cred-b8c3e972a39f: secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:09.357962 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.353129    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9: secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:09.357962 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.353266    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials podName:7fcfb069-7210-45e2-a830-5b4294e0ab20 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:09.853238415 +0000 UTC m=+2693.758092848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20") : secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:09.357962 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.343080    5393 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-plugins-e2e-extmod-b-5711cf: configmap "payload-processing-external-model-plugins-e2e-extmod-b-5711cf" not found
Nov 03 11:14:09.357962 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.353330    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume podName:7fcfb069-7210-45e2-a830-5b4294e0ab20 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:09.853319169 +0000 UTC m=+2693.758173606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-5711cf" not found
Nov 03 11:14:09.394006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:09.393937    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" podStartSLOduration=2.393900819 podStartE2EDuration="2.393900819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:09.393270159 +0000 UTC m=+2693.298124602" watchObservedRunningTime="2026-11-03 11:14:09.393900819 +0000 UTC m=+2693.298755261"
Nov 03 11:14:09.953886 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.953854    5393 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-plugins-e2e-extmod-b-5711cf: configmap "payload-processing-external-model-plugins-e2e-extmod-b-5711cf" not found
Nov 03 11:14:09.954115 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.953853    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5711cf/agc-cred-b8c3e972a39f: secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:09.954115 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.953960    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9: secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:09.954115 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.953968    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume podName:7fcfb069-7210-45e2-a830-5b4294e0ab20 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:10.953908241 +0000 UTC m=+2694.858762661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-5711cf" not found
Nov 03 11:14:09.954115 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:09.954030    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials podName:7fcfb069-7210-45e2-a830-5b4294e0ab20 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:10.954011649 +0000 UTC m=+2694.858866083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20") : secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:10.980332 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:10.980299    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5711cf/agc-cred-b8c3e972a39f: secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:10.980332 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:10.980315    5393 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-plugins-e2e-extmod-b-5711cf: configmap "payload-processing-external-model-plugins-e2e-extmod-b-5711cf" not found
Nov 03 11:14:10.980332 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:10.980327    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9: secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:10.980986 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:10.980394    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume podName:7fcfb069-7210-45e2-a830-5b4294e0ab20 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:12.980374071 +0000 UTC m=+2696.885228494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-5711cf" not found
Nov 03 11:14:10.980986 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:10.980412    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials podName:7fcfb069-7210-45e2-a830-5b4294e0ab20 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:12.980402609 +0000 UTC m=+2696.885257028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20") : secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:13.018047 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:13.018015    5393 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-plugins-e2e-extmod-b-5711cf: configmap "payload-processing-external-model-plugins-e2e-extmod-b-5711cf" not found
Nov 03 11:14:13.018047 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:13.018027    5393 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5711cf/agc-cred-b8c3e972a39f: secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:13.018047 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:13.018052    5393 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9: secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:13.018540 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:13.018088    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume podName:7fcfb069-7210-45e2-a830-5b4294e0ab20 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:17.018072749 +0000 UTC m=+2700.922927168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-5711cf" not found
Nov 03 11:14:13.018540 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:13.018105    5393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials podName:7fcfb069-7210-45e2-a830-5b4294e0ab20 nodeName:}" failed. No retries permitted until 2026-11-03 11:14:17.018098285 +0000 UTC m=+2700.922952704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20") : secret "agc-cred-b8c3e972a39f" not found
Nov 03 11:14:13.192578 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:13.192535    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:14:15.319634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.319585    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" podUID="7fcfb069-7210-45e2-a830-5b4294e0ab20" containerName="payload-processing" containerID="cri-o://b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14" gracePeriod=25
Nov 03 11:14:15.645022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.644997    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:15.783673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.783632    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-kube-api-access-9xxf5\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-kube-api-access-9xxf5\") pod \"7fcfb069-7210-45e2-a830-5b4294e0ab20\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") "
Nov 03 11:14:15.783862 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.783761    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-routing-overlay-volume\") pod \"7fcfb069-7210-45e2-a830-5b4294e0ab20\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") "
Nov 03 11:14:15.783862 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.783836    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume\") pod \"7fcfb069-7210-45e2-a830-5b4294e0ab20\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") "
Nov 03 11:14:15.784132 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.784111    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials\") pod \"7fcfb069-7210-45e2-a830-5b4294e0ab20\" (UID: \"7fcfb069-7210-45e2-a830-5b4294e0ab20\") "
Nov 03 11:14:15.784228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.784192    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-routing-overlay-volume" pod "7fcfb069-7210-45e2-a830-5b4294e0ab20" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:15.784377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.784350    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume" pod "7fcfb069-7210-45e2-a830-5b4294e0ab20" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:15.784986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.784959    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:15.785100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.784991    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcfb069-7210-45e2-a830-5b4294e0ab20-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:15.787058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.786986    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-kube-api-access-9xxf5" pod "7fcfb069-7210-45e2-a830-5b4294e0ab20" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20"). InnerVolumeSpecName "kube-api-access-9xxf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:15.787058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.787034    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials" pod "7fcfb069-7210-45e2-a830-5b4294e0ab20" (UID: "7fcfb069-7210-45e2-a830-5b4294e0ab20"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:15.887050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.886947    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:15.887050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:15.886983    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xxf5\" (UniqueName: \"kubernetes.io/projected/7fcfb069-7210-45e2-a830-5b4294e0ab20-kube-api-access-9xxf5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:16.212431 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.212330    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:16.410428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.410394    5393 generic.go:356] "Generic (PLEG): container finished" podID="7fcfb069-7210-45e2-a830-5b4294e0ab20" containerID="b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14" exitCode=0
Nov 03 11:14:16.417301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.417250    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" event={"ID":"7fcfb069-7210-45e2-a830-5b4294e0ab20","Type":"ContainerDied","Data":"b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14"}
Nov 03 11:14:16.417646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.417617    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9" event={"ID":"7fcfb069-7210-45e2-a830-5b4294e0ab20","Type":"ContainerDied","Data":"e863d517002d958fd0bc26eb7be20e43e3420eac01710d5ed5ea2b7112ef38a3"}
Nov 03 11:14:16.417808 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.417785    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"
Nov 03 11:14:16.434970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.417788    5393 scope.go:122] "RemoveContainer" containerID="b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14"
Nov 03 11:14:16.498196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.494017    5393 scope.go:122] "RemoveContainer" containerID="b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14"
Nov 03 11:14:16.526048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.526000    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"]
Nov 03 11:14:16.526372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.526356    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:16.539107 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:16.539060    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14\": container with ID starting with b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14 not found: ID does not exist" containerID="b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14"
Nov 03 11:14:16.539349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.539322    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14"} err="failed to get container status \"b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14\": rpc error: code = NotFound desc = could not find container \"b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14\": container with ID starting with b7ec742309ab380512c9bd664da06a6e3021a8e7ebaa903ed2c13fb1abd72b14 not found: ID does not exist"
Nov 03 11:14:16.552099 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.544512    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5711cf/payload-processing-external-model-e2e-extmod-b-5711cf-8458znrk9"]
Nov 03 11:14:16.560377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:16.560348    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:14:18.317573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:18.317537    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fcfb069-7210-45e2-a830-5b4294e0ab20" path="/var/lib/kubelet/pods/7fcfb069-7210-45e2-a830-5b4294e0ab20/volumes"
Nov 03 11:14:18.356508 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:18.356478    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:14:18.357135 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:18.357112    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:14:20.104200 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:20.104162    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:20.268713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:20.268677    5393 scope.go:122] "RemoveContainer" containerID="8141c2173a0003312a41e9030930601cb4391afbda4891f423947a12df061bf6"
Nov 03 11:14:20.701419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:20.701383    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"]
Nov 03 11:14:21.572104 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.572060    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"]
Nov 03 11:14:21.573188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.573163    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7fcfb069-7210-45e2-a830-5b4294e0ab20" containerName="payload-processing"
Nov 03 11:14:21.573188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.573188    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fcfb069-7210-45e2-a830-5b4294e0ab20" containerName="payload-processing"
Nov 03 11:14:21.573412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.573397    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fcfb069-7210-45e2-a830-5b4294e0ab20" containerName="payload-processing"
Nov 03 11:14:21.582196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.582167    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:21.586617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.586588    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"]
Nov 03 11:14:21.587420 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.587395    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a68103\""
Nov 03 11:14:21.587575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.587454    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a68103-dockercfg-t696b\""
Nov 03 11:14:21.842997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.839471    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed16f6d6-d892-4603-a936-40f5e7ca4e73-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc\" (UID: \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:21.842997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.839721    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4s6\" (UniqueName: \"kubernetes.io/projected/ed16f6d6-d892-4603-a936-40f5e7ca4e73-kube-api-access-fs4s6\") pod \"payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc\" (UID: \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:21.950608 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.950564    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed16f6d6-d892-4603-a936-40f5e7ca4e73-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc\" (UID: \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:21.962120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.961893    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4s6\" (UniqueName: \"kubernetes.io/projected/ed16f6d6-d892-4603-a936-40f5e7ca4e73-kube-api-access-fs4s6\") pod \"payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc\" (UID: \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:21.964586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.964150    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed16f6d6-d892-4603-a936-40f5e7ca4e73-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc\" (UID: \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:21.987947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:21.980060    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4s6\" (UniqueName: \"kubernetes.io/projected/ed16f6d6-d892-4603-a936-40f5e7ca4e73-kube-api-access-fs4s6\") pod \"payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc\" (UID: \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:22.200658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:22.200623    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:22.530785 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:22.530751    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"]
Nov 03 11:14:22.532048 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:14:22.531387    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded16f6d6_d892_4603_a936_40f5e7ca4e73.slice/crio-73725ce67a309ed6f828799f3e44468269a97798a2a8cfd886e304cf25b6a6d0 WatchSource:0}: Error finding container 73725ce67a309ed6f828799f3e44468269a97798a2a8cfd886e304cf25b6a6d0: Status 404 returned error can't find the container with id 73725ce67a309ed6f828799f3e44468269a97798a2a8cfd886e304cf25b6a6d0
Nov 03 11:14:23.524152 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:23.524119    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc" event={"ID":"ed16f6d6-d892-4603-a936-40f5e7ca4e73","Type":"ContainerStarted","Data":"48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7"}
Nov 03 11:14:23.524666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:23.524178    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc" event={"ID":"ed16f6d6-d892-4603-a936-40f5e7ca4e73","Type":"ContainerStarted","Data":"73725ce67a309ed6f828799f3e44468269a97798a2a8cfd886e304cf25b6a6d0"}
Nov 03 11:14:23.524666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:23.524224    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:23.541154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:23.541097    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc" podStartSLOduration=2.541078426 podStartE2EDuration="2.541078426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:23.539346066 +0000 UTC m=+2707.444200510" watchObservedRunningTime="2026-11-03 11:14:23.541078426 +0000 UTC m=+2707.445932874"
Nov 03 11:14:25.702151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:25.702101    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m" podUID="bc05d406-c2f5-489f-81e9-c9b32339b6e8" containerName="payload-processing" containerID="cri-o://a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db" gracePeriod=25
Nov 03 11:14:26.071076 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.071048    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:14:26.161778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.161735    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-plugins-config-volume\") pod \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") "
Nov 03 11:14:26.161778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.161783    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-provider-credentials\") pod \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") "
Nov 03 11:14:26.162038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.161895    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-routing-overlay-volume\") pod \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") "
Nov 03 11:14:26.162038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.161997    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-kube-api-access-cfq98\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-kube-api-access-cfq98\") pod \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\" (UID: \"bc05d406-c2f5-489f-81e9-c9b32339b6e8\") "
Nov 03 11:14:26.162274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.162244    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-routing-overlay-volume" pod "bc05d406-c2f5-489f-81e9-c9b32339b6e8" (UID: "bc05d406-c2f5-489f-81e9-c9b32339b6e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:26.162274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.162264    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-plugins-config-volume" pod "bc05d406-c2f5-489f-81e9-c9b32339b6e8" (UID: "bc05d406-c2f5-489f-81e9-c9b32339b6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:26.162725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.162704    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:26.162798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.162731    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc05d406-c2f5-489f-81e9-c9b32339b6e8-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:26.164539 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.164509    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-provider-credentials" pod "bc05d406-c2f5-489f-81e9-c9b32339b6e8" (UID: "bc05d406-c2f5-489f-81e9-c9b32339b6e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:26.164694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.164667    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-kube-api-access-cfq98" pod "bc05d406-c2f5-489f-81e9-c9b32339b6e8" (UID: "bc05d406-c2f5-489f-81e9-c9b32339b6e8"). InnerVolumeSpecName "kube-api-access-cfq98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:26.264127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.264101    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:26.264127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.264127    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfq98\" (UniqueName: \"kubernetes.io/projected/bc05d406-c2f5-489f-81e9-c9b32339b6e8-kube-api-access-cfq98\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:26.551645 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.551608    5393 generic.go:356] "Generic (PLEG): container finished" podID="bc05d406-c2f5-489f-81e9-c9b32339b6e8" containerID="a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db" exitCode=0
Nov 03 11:14:26.551854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.551662    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"
Nov 03 11:14:26.551854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.551683    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m" event={"ID":"bc05d406-c2f5-489f-81e9-c9b32339b6e8","Type":"ContainerDied","Data":"a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db"}
Nov 03 11:14:26.551854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.551736    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m" event={"ID":"bc05d406-c2f5-489f-81e9-c9b32339b6e8","Type":"ContainerDied","Data":"c9f7e6b6ccd45cd47a1bf273f5fc307db55196fa3c6a1b75c0e24d25f6d7420f"}
Nov 03 11:14:26.551854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.551756    5393 scope.go:122] "RemoveContainer" containerID="a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db"
Nov 03 11:14:26.563838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.563809    5393 scope.go:122] "RemoveContainer" containerID="a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db"
Nov 03 11:14:26.564160 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:26.564140    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db\": container with ID starting with a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db not found: ID does not exist" containerID="a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db"
Nov 03 11:14:26.564235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.564169    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db"} err="failed to get container status \"a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db\": rpc error: code = NotFound desc = could not find container \"a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db\": container with ID starting with a3ec9353ffc4cf119880bab9cdff5aa2ce2ab8a1df03f2dc1f98690538fc60db not found: ID does not exist"
Nov 03 11:14:26.570903 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.570881    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"]
Nov 03 11:14:26.575216 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:26.575192    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9f53bb/payload-processing-external-model-e2e-extmod-a-9f53bb-d8d84sq2m"]
Nov 03 11:14:28.233804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:28.233772    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc05d406-c2f5-489f-81e9-c9b32339b6e8" path="/var/lib/kubelet/pods/bc05d406-c2f5-489f-81e9-c9b32339b6e8/volumes"
Nov 03 11:14:29.027785 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:29.027728    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"]
Nov 03 11:14:29.754690 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:29.754660    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:30.126104 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:30.126066    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:32.130343 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:32.130305    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:32.130831 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:32.130378    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:34.028769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.028700    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf" podUID="fc68a473-21f6-4b51-ae17-bbd1a5effcc4" containerName="payload-processing" containerID="cri-o://c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f" gracePeriod=25
Nov 03 11:14:34.330525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.330501    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:14:34.409698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.409659    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-plugins-config-volume\") pod \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") "
Nov 03 11:14:34.409935 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.409726    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-kube-api-access-7h6ql\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-kube-api-access-7h6ql\") pod \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") "
Nov 03 11:14:34.410418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.410048    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-provider-credentials\") pod \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") "
Nov 03 11:14:34.410418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.410140    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-routing-overlay-volume\") pod \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\" (UID: \"fc68a473-21f6-4b51-ae17-bbd1a5effcc4\") "
Nov 03 11:14:34.410418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.410136    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-plugins-config-volume" pod "fc68a473-21f6-4b51-ae17-bbd1a5effcc4" (UID: "fc68a473-21f6-4b51-ae17-bbd1a5effcc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:34.410660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.410455    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-routing-overlay-volume" pod "fc68a473-21f6-4b51-ae17-bbd1a5effcc4" (UID: "fc68a473-21f6-4b51-ae17-bbd1a5effcc4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:34.411155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.411133    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:34.411231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.411165    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:34.412273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.412241    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-kube-api-access-7h6ql" pod "fc68a473-21f6-4b51-ae17-bbd1a5effcc4" (UID: "fc68a473-21f6-4b51-ae17-bbd1a5effcc4"). InnerVolumeSpecName "kube-api-access-7h6ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:34.412464 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.412448    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-provider-credentials" pod "fc68a473-21f6-4b51-ae17-bbd1a5effcc4" (UID: "fc68a473-21f6-4b51-ae17-bbd1a5effcc4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:34.512879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.512845    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6ql\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-kube-api-access-7h6ql\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:34.512879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.512875    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc68a473-21f6-4b51-ae17-bbd1a5effcc4-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:34.536157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.536128    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:14:34.693995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.693961    5393 generic.go:356] "Generic (PLEG): container finished" podID="fc68a473-21f6-4b51-ae17-bbd1a5effcc4" containerID="c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f" exitCode=0
Nov 03 11:14:34.694181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.694012    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"
Nov 03 11:14:34.694181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.694043    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf" event={"ID":"fc68a473-21f6-4b51-ae17-bbd1a5effcc4","Type":"ContainerDied","Data":"c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f"}
Nov 03 11:14:34.694181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.694102    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf" event={"ID":"fc68a473-21f6-4b51-ae17-bbd1a5effcc4","Type":"ContainerDied","Data":"e3e7a384cc2cdc3c6414843c48c06ee7a9b03814ef55b9b85e0cdac9db98d0ec"}
Nov 03 11:14:34.694181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.694125    5393 scope.go:122] "RemoveContainer" containerID="c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f"
Nov 03 11:14:34.706545 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.706519    5393 scope.go:122] "RemoveContainer" containerID="c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f"
Nov 03 11:14:34.706935 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:34.706900    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f\": container with ID starting with c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f not found: ID does not exist" containerID="c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f"
Nov 03 11:14:34.707025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.706967    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f"} err="failed to get container status \"c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f\": rpc error: code = NotFound desc = could not find container \"c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f\": container with ID starting with c714a209b47542abea93e3b3ff1e063965980b5cd6eb8f2d87bf9bc301686f3f not found: ID does not exist"
Nov 03 11:14:34.719026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.719000    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"]
Nov 03 11:14:34.724265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:34.724243    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-7bd775srf"]
Nov 03 11:14:36.232565 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:36.232535    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc68a473-21f6-4b51-ae17-bbd1a5effcc4" path="/var/lib/kubelet/pods/fc68a473-21f6-4b51-ae17-bbd1a5effcc4/volumes"
Nov 03 11:14:38.156941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.151769    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"]
Nov 03 11:14:38.190946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.172680    5393 cpu_manager.go:414] "containerMap: removing container" podUID="fc68a473-21f6-4b51-ae17-bbd1a5effcc4" containerName="payload-processing"
Nov 03 11:14:38.190946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.172714    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc68a473-21f6-4b51-ae17-bbd1a5effcc4" containerName="payload-processing"
Nov 03 11:14:38.190946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.172752    5393 cpu_manager.go:414] "containerMap: removing container" podUID="bc05d406-c2f5-489f-81e9-c9b32339b6e8" containerName="payload-processing"
Nov 03 11:14:38.190946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.172759    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc05d406-c2f5-489f-81e9-c9b32339b6e8" containerName="payload-processing"
Nov 03 11:14:38.190946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.172962    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc68a473-21f6-4b51-ae17-bbd1a5effcc4" containerName="payload-processing"
Nov 03 11:14:38.190946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.172979    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc05d406-c2f5-489f-81e9-c9b32339b6e8" containerName="payload-processing"
Nov 03 11:14:38.351956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.319468    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.385401 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.354535    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-917da2\""
Nov 03 11:14:38.385809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.385768    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"agc-cred-08e898657fc1\""
Nov 03 11:14:38.403194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.366671    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"kube-root-ca.crt\""
Nov 03 11:14:38.403436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.390126    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"openshift-service-ca.crt\""
Nov 03 11:14:38.403554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.390226    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"payload-processing-external-model-e2e--90f9edc7-dockercfg-87cr7\""
Nov 03 11:14:38.403660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.366585    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-917da2\"/\"routing-overlay\""
Nov 03 11:14:38.589070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.562661    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.592956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.592372    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.592956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.592650    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62btf\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-kube-api-access-62btf\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.592956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.592763    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.640970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.639013    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"]
Nov 03 11:14:38.834434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.805249    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62btf\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-kube-api-access-62btf\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.834434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.805349    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.834434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.805429    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.834434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.805640    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.834434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.806596    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.834434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.807616    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:38.834434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.808284    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:14:38.933265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.888331    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:14:38.962372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.962344    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:14:39.063587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:38.962132    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 11:14:39.065473 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.028703    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"]
Nov 03 11:14:39.089003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.042624    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 11:14:39.089862 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.042747    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.152571    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62btf\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-kube-api-access-62btf\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.153084    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") " pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.154859    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.155609    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"]
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.155691    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.160407    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"agc-cred-8b132e46c538\""
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.160514    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"openshift-service-ca.crt\""
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.160555    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-e2e--b8bff1be-dockercfg-nsr7g\""
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.160561    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"kube-root-ca.crt\""
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.160607    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fb5766\""
Nov 03 11:14:39.160858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.160635    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fb5766\"/\"routing-overlay\""
Nov 03 11:14:39.225858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.225824    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.226081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.225945    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqfr\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-kube-api-access-ltqfr\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.226081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.226032    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.226081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.226064    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.327510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.327475    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.327707 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.327620    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqfr\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-kube-api-access-ltqfr\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.327707 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.327689    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.327846 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.327718    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.328721 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.328693    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.329026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.328997    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.330886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.330863    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.336288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.336263    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqfr\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-kube-api-access-ltqfr\") pod \"payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") " pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.479042 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.479009    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"]
Nov 03 11:14:39.532371 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.531715    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:39.535939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:39.535878    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"]
Nov 03 11:14:40.131934 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:40.111498    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"]
Nov 03 11:14:40.135314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:40.135283    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"]
Nov 03 11:14:40.146983 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:40.146946    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk" event={"ID":"831316af-a3d5-4c48-9fc6-31f11f1e378e","Type":"ContainerStarted","Data":"fa22e02beec8b50792f4b19998d422b130987ec78f3dbade9b5084a0455a5ab9"}
Nov 03 11:14:40.147151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:40.147048    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk" event={"ID":"831316af-a3d5-4c48-9fc6-31f11f1e378e","Type":"ContainerStarted","Data":"e2d3b78222b20e13f72e5887be1feab08d427d190545d512185009e0dc3d12a6"}
Nov 03 11:14:40.147725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:40.147405    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:40.333942 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:40.330820    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:40.482486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:40.482419    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk" podStartSLOduration=2.482397235 podStartE2EDuration="2.482397235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:40.469342998 +0000 UTC m=+2724.374197441" watchObservedRunningTime="2026-11-03 11:14:40.482397235 +0000 UTC m=+2724.387251680"
Nov 03 11:14:41.157883 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:41.157844    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn" event={"ID":"c0cca15d-75ff-4325-93b0-c1b242eb939f","Type":"ContainerStarted","Data":"271addc8888743159ecda7c171457cae0176966d8e98c09cdfc03c7f440f9310"}
Nov 03 11:14:41.158104 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:41.157945    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:41.158104 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:41.157973    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn" event={"ID":"c0cca15d-75ff-4325-93b0-c1b242eb939f","Type":"ContainerStarted","Data":"cb07165f473c90158e944e5c946405aefe39111f21b630727e473a923d92df78"}
Nov 03 11:14:41.177677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:41.177628    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn" podStartSLOduration=3.177613222 podStartE2EDuration="3.177613222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:41.174403659 +0000 UTC m=+2725.079258101" watchObservedRunningTime="2026-11-03 11:14:41.177613222 +0000 UTC m=+2725.082467660"
Nov 03 11:14:43.391170 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:43.391121    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:14:44.466272 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:44.466237    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:45.147969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.147889    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk" podUID="831316af-a3d5-4c48-9fc6-31f11f1e378e" containerName="payload-processing" containerID="cri-o://fa22e02beec8b50792f4b19998d422b130987ec78f3dbade9b5084a0455a5ab9" gracePeriod=25
Nov 03 11:14:45.328632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.328594    5393 generic.go:356] "Generic (PLEG): container finished" podID="831316af-a3d5-4c48-9fc6-31f11f1e378e" containerID="fa22e02beec8b50792f4b19998d422b130987ec78f3dbade9b5084a0455a5ab9" exitCode=0
Nov 03 11:14:45.328837 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.328679    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk" event={"ID":"831316af-a3d5-4c48-9fc6-31f11f1e378e","Type":"ContainerDied","Data":"fa22e02beec8b50792f4b19998d422b130987ec78f3dbade9b5084a0455a5ab9"}
Nov 03 11:14:45.450734 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.450704    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:45.513958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.513900    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-kube-api-access-62btf\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-kube-api-access-62btf\") pod \"831316af-a3d5-4c48-9fc6-31f11f1e378e\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") "
Nov 03 11:14:45.514699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.514669    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-plugins-config-volume\") pod \"831316af-a3d5-4c48-9fc6-31f11f1e378e\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") "
Nov 03 11:14:45.515037 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.515012    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-routing-overlay-volume\") pod \"831316af-a3d5-4c48-9fc6-31f11f1e378e\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") "
Nov 03 11:14:45.515160 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.515133    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-plugins-config-volume" pod "831316af-a3d5-4c48-9fc6-31f11f1e378e" (UID: "831316af-a3d5-4c48-9fc6-31f11f1e378e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:45.515247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.515205    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-provider-credentials\") pod \"831316af-a3d5-4c48-9fc6-31f11f1e378e\" (UID: \"831316af-a3d5-4c48-9fc6-31f11f1e378e\") "
Nov 03 11:14:45.516101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.515622    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-routing-overlay-volume" pod "831316af-a3d5-4c48-9fc6-31f11f1e378e" (UID: "831316af-a3d5-4c48-9fc6-31f11f1e378e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:45.516101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.516052    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:45.516101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.516077    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/831316af-a3d5-4c48-9fc6-31f11f1e378e-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:45.517872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.517846    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-kube-api-access-62btf" pod "831316af-a3d5-4c48-9fc6-31f11f1e378e" (UID: "831316af-a3d5-4c48-9fc6-31f11f1e378e"). InnerVolumeSpecName "kube-api-access-62btf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:45.518154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.518128    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-provider-credentials" pod "831316af-a3d5-4c48-9fc6-31f11f1e378e" (UID: "831316af-a3d5-4c48-9fc6-31f11f1e378e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:45.617879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.617837    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:45.617879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:45.617872    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62btf\" (UniqueName: \"kubernetes.io/projected/831316af-a3d5-4c48-9fc6-31f11f1e378e-kube-api-access-62btf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:46.165491 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.163998    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn" podUID="c0cca15d-75ff-4325-93b0-c1b242eb939f" containerName="payload-processing" containerID="cri-o://271addc8888743159ecda7c171457cae0176966d8e98c09cdfc03c7f440f9310" gracePeriod=25
Nov 03 11:14:46.370105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.364023    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk" event={"ID":"831316af-a3d5-4c48-9fc6-31f11f1e378e","Type":"ContainerDied","Data":"e2d3b78222b20e13f72e5887be1feab08d427d190545d512185009e0dc3d12a6"}
Nov 03 11:14:46.370105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.364140    5393 scope.go:122] "RemoveContainer" containerID="fa22e02beec8b50792f4b19998d422b130987ec78f3dbade9b5084a0455a5ab9"
Nov 03 11:14:46.370105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.364309    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"
Nov 03 11:14:46.375130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.374952    5393 generic.go:356] "Generic (PLEG): container finished" podID="c0cca15d-75ff-4325-93b0-c1b242eb939f" containerID="271addc8888743159ecda7c171457cae0176966d8e98c09cdfc03c7f440f9310" exitCode=0
Nov 03 11:14:46.375130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.375022    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn" event={"ID":"c0cca15d-75ff-4325-93b0-c1b242eb939f","Type":"ContainerDied","Data":"271addc8888743159ecda7c171457cae0176966d8e98c09cdfc03c7f440f9310"}
Nov 03 11:14:46.411797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.411763    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"]
Nov 03 11:14:46.417134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.417072    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-917da2/payload-processing-external-model-e2e-extmod-a-917da2-d6f6k87wk"]
Nov 03 11:14:46.583275 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.583242    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:46.649295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.649258    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-provider-credentials\") pod \"c0cca15d-75ff-4325-93b0-c1b242eb939f\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") "
Nov 03 11:14:46.649578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.649358    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-routing-overlay-volume\") pod \"c0cca15d-75ff-4325-93b0-c1b242eb939f\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") "
Nov 03 11:14:46.649578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.649402    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-kube-api-access-ltqfr\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-kube-api-access-ltqfr\") pod \"c0cca15d-75ff-4325-93b0-c1b242eb939f\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") "
Nov 03 11:14:46.649811 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.649773    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-plugins-config-volume\") pod \"c0cca15d-75ff-4325-93b0-c1b242eb939f\" (UID: \"c0cca15d-75ff-4325-93b0-c1b242eb939f\") "
Nov 03 11:14:46.650269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.650242    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-routing-overlay-volume" pod "c0cca15d-75ff-4325-93b0-c1b242eb939f" (UID: "c0cca15d-75ff-4325-93b0-c1b242eb939f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:46.651633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.650789    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:46.651633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.650978    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-plugins-config-volume" pod "c0cca15d-75ff-4325-93b0-c1b242eb939f" (UID: "c0cca15d-75ff-4325-93b0-c1b242eb939f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:14:46.653410 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.653380    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-provider-credentials" pod "c0cca15d-75ff-4325-93b0-c1b242eb939f" (UID: "c0cca15d-75ff-4325-93b0-c1b242eb939f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:46.654416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.654390    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-kube-api-access-ltqfr" pod "c0cca15d-75ff-4325-93b0-c1b242eb939f" (UID: "c0cca15d-75ff-4325-93b0-c1b242eb939f"). InnerVolumeSpecName "kube-api-access-ltqfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:14:46.752376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.752340    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0cca15d-75ff-4325-93b0-c1b242eb939f-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:46.752376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.752370    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:46.752376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:46.752379    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltqfr\" (UniqueName: \"kubernetes.io/projected/c0cca15d-75ff-4325-93b0-c1b242eb939f-kube-api-access-ltqfr\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:14:47.429944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.429406    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn" event={"ID":"c0cca15d-75ff-4325-93b0-c1b242eb939f","Type":"ContainerDied","Data":"cb07165f473c90158e944e5c946405aefe39111f21b630727e473a923d92df78"}
Nov 03 11:14:47.429944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.429487    5393 scope.go:122] "RemoveContainer" containerID="271addc8888743159ecda7c171457cae0176966d8e98c09cdfc03c7f440f9310"
Nov 03 11:14:47.429944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.429624    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"
Nov 03 11:14:47.495465 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.495416    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"]
Nov 03 11:14:47.510284 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.510255    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fb5766/payload-processing-external-model-e2e-extmod-b-fb5766-8d666xtjn"]
Nov 03 11:14:47.646230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.643700    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"]
Nov 03 11:14:47.646230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.645008    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c0cca15d-75ff-4325-93b0-c1b242eb939f" containerName="payload-processing"
Nov 03 11:14:47.646230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.645031    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cca15d-75ff-4325-93b0-c1b242eb939f" containerName="payload-processing"
Nov 03 11:14:47.646230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.645089    5393 cpu_manager.go:414] "containerMap: removing container" podUID="831316af-a3d5-4c48-9fc6-31f11f1e378e" containerName="payload-processing"
Nov 03 11:14:47.646230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.645096    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831316af-a3d5-4c48-9fc6-31f11f1e378e" containerName="payload-processing"
Nov 03 11:14:47.646230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.645292    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cca15d-75ff-4325-93b0-c1b242eb939f" containerName="payload-processing"
Nov 03 11:14:47.646230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.645321    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="831316af-a3d5-4c48-9fc6-31f11f1e378e" containerName="payload-processing"
Nov 03 11:14:47.653290 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.652070    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.659122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.659096    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-055d82-openshift-default-dockercfg-fpkbq\""
Nov 03 11:14:47.660016 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.659718    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"]
Nov 03 11:14:47.711570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.711472    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.711766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.711587    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2310479c-98fb-4cc8-8b03-59003bd96943-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.711766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.711628    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.711989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.711966    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.712093 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.712008    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.712155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.712131    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.712219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.712162    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.712219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.712188    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2310479c-98fb-4cc8-8b03-59003bd96943-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.712219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.712215    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gpt\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-kube-api-access-w6gpt\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.865686    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.865813    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2310479c-98fb-4cc8-8b03-59003bd96943-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.865859    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.866141    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.866178    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.866248    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.866272    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.866295    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2310479c-98fb-4cc8-8b03-59003bd96943-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.866660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.866319    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gpt\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-kube-api-access-w6gpt\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.889945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.885269    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.889945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.885551    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.936951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.896541    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.936951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.896813    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.936951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.908013    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2310479c-98fb-4cc8-8b03-59003bd96943-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.936951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.910749    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gpt\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-kube-api-access-w6gpt\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.949943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.945432    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.951251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.951221    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:47.982640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:47.982540    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2310479c-98fb-4cc8-8b03-59003bd96943-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-bln7k\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:48.233657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:48.233573    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831316af-a3d5-4c48-9fc6-31f11f1e378e" path="/var/lib/kubelet/pods/831316af-a3d5-4c48-9fc6-31f11f1e378e/volumes"
Nov 03 11:14:48.234134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:48.234119    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cca15d-75ff-4325-93b0-c1b242eb939f" path="/var/lib/kubelet/pods/c0cca15d-75ff-4325-93b0-c1b242eb939f/volumes"
Nov 03 11:14:48.275661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:48.275628    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:48.572130 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:14:48.572101    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2310479c_98fb_4cc8_8b03_59003bd96943.slice/crio-8514c96afbb0fda37d833d3f4f8e63de2ef31d06e71e4be6522e042490a1fce3 WatchSource:0}: Error finding container 8514c96afbb0fda37d833d3f4f8e63de2ef31d06e71e4be6522e042490a1fce3: Status 404 returned error can't find the container with id 8514c96afbb0fda37d833d3f4f8e63de2ef31d06e71e4be6522e042490a1fce3
Nov 03 11:14:48.572277 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:48.572166    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"]
Nov 03 11:14:48.575222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:48.575190    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:48.575336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:48.575267    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:48.575336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:48.575311    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:14:49.461231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:49.461188    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" event={"ID":"2310479c-98fb-4cc8-8b03-59003bd96943","Type":"ContainerStarted","Data":"64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d"}
Nov 03 11:14:49.461661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:49.461292    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" event={"ID":"2310479c-98fb-4cc8-8b03-59003bd96943","Type":"ContainerStarted","Data":"8514c96afbb0fda37d833d3f4f8e63de2ef31d06e71e4be6522e042490a1fce3"}
Nov 03 11:14:49.483680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:49.482889    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podStartSLOduration=2.482874894 podStartE2EDuration="2.482874894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:49.47981012 +0000 UTC m=+2733.384664564" watchObservedRunningTime="2026-11-03 11:14:49.482874894 +0000 UTC m=+2733.387729334"
Nov 03 11:14:50.276947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:50.276724    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:50.287569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:50.287535    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:50.469692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:50.469661    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:50.470692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:50.470665    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:14:50.575239 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:50.575160    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:14:53.085585 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.085553    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"]
Nov 03 11:14:53.090970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.090908    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:53.097649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.097613    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-055d82\""
Nov 03 11:14:53.098717 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.098689    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-055d82-dockercfg-4dgcg\""
Nov 03 11:14:53.099963 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.099898    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"]
Nov 03 11:14:53.239941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.239239    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252\" (UID: \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:53.239941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.239373    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xns\" (UniqueName: \"kubernetes.io/projected/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-kube-api-access-n8xns\") pod \"payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252\" (UID: \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:53.402383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.402336    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252\" (UID: \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:53.466976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.462365    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xns\" (UniqueName: \"kubernetes.io/projected/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-kube-api-access-n8xns\") pod \"payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252\" (UID: \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:53.514933 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.514881    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252\" (UID: \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:53.535120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.535074    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xns\" (UniqueName: \"kubernetes.io/projected/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-kube-api-access-n8xns\") pod \"payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252\" (UID: \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:53.712582 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:53.712497    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:54.004853 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:54.004822    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"]
Nov 03 11:14:54.004989 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:14:54.004828    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30acda02_b9e6_4bdf_a3b4_5f82b0ef4024.slice/crio-efba5850d74bfd3a7f7bc6d4665b1f46490ba19d4675b4f72454e5c3afa910f2 WatchSource:0}: Error finding container efba5850d74bfd3a7f7bc6d4665b1f46490ba19d4675b4f72454e5c3afa910f2: Status 404 returned error can't find the container with id efba5850d74bfd3a7f7bc6d4665b1f46490ba19d4675b4f72454e5c3afa910f2
Nov 03 11:14:54.543586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:54.543557    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252" event={"ID":"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024","Type":"ContainerStarted","Data":"f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711"}
Nov 03 11:14:54.544056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:54.543610    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252" event={"ID":"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024","Type":"ContainerStarted","Data":"efba5850d74bfd3a7f7bc6d4665b1f46490ba19d4675b4f72454e5c3afa910f2"}
Nov 03 11:14:54.544056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:54.543714    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:14:54.560949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:14:54.560888    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252" podStartSLOduration=1.560874485 podStartE2EDuration="1.560874485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:14:54.558900267 +0000 UTC m=+2738.463754707" watchObservedRunningTime="2026-11-03 11:14:54.560874485 +0000 UTC m=+2738.465728931"
Nov 03 11:14:59.493348 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:14:59.493272    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:15:00.139573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.139539    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"]
Nov 03 11:15:00.144865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.144843    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"]
Nov 03 11:15:00.145003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.144991    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:00.149860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.149832    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:00.171288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.171259    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"]
Nov 03 11:15:00.176398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.176370    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"]
Nov 03 11:15:00.186261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.186220    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"]
Nov 03 11:15:00.194667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.194633    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:00.195995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.195967    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"]
Nov 03 11:15:00.217594 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.215357    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"]
Nov 03 11:15:00.218875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.218569    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:00.237528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.237499    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:00.257009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.254306    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:15:00.272744 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.272716    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"]
Nov 03 11:15:00.283965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.283464    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"]
Nov 03 11:15:00.290663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.290636    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"]
Nov 03 11:15:00.296396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.296370    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:00.297492 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.297471    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"]
Nov 03 11:15:00.304006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.303988    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:00.316356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.316326    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"]
Nov 03 11:15:00.325467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.325417    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"]
Nov 03 11:15:00.330396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.330371    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"]
Nov 03 11:15:00.334677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.334651    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"]
Nov 03 11:15:00.335841 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.335812    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf3819-7aea-4e1c-ad10-66f59268dcf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl\" (UID: \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:00.335986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.335872    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bdhq\" (UniqueName: \"kubernetes.io/projected/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-kube-api-access-9bdhq\") pod \"maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6\" (UID: \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:00.336272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336253    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp7n\" (UniqueName: \"kubernetes.io/projected/26324c17-9964-4c55-8f6c-30c6511f7048-kube-api-access-whp7n\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9\" (UID: \"26324c17-9964-4c55-8f6c-30c6511f7048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:00.336325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336308    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1107f46f-c86c-4569-b755-9aa28e256af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n\" (UID: \"1107f46f-c86c-4569-b755-9aa28e256af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:00.336378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336331    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-542lf\" (UniqueName: \"kubernetes.io/projected/2e754a45-81bc-45e1-9949-5c8c9f6a638c-kube-api-access-542lf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk\" (UID: \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:00.336419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336389    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5\" (UID: \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:00.336466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336457    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xrx\" (UniqueName: \"kubernetes.io/projected/1107f46f-c86c-4569-b755-9aa28e256af0-kube-api-access-99xrx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n\" (UID: \"1107f46f-c86c-4569-b755-9aa28e256af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:00.336502 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336477    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e754a45-81bc-45e1-9949-5c8c9f6a638c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk\" (UID: \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:00.336537 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336527    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6\" (UID: \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:00.336573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336549    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt662\" (UniqueName: \"kubernetes.io/projected/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-kube-api-access-qt662\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5\" (UID: \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:00.336614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336584    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr\" (UID: \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:00.336656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336613    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppwj\" (UniqueName: \"kubernetes.io/projected/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-kube-api-access-sppwj\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr\" (UID: \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:00.336694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336656    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5dgr\" (UniqueName: \"kubernetes.io/projected/36bf3819-7aea-4e1c-ad10-66f59268dcf1-kube-api-access-s5dgr\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl\" (UID: \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:00.336694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.336677    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26324c17-9964-4c55-8f6c-30c6511f7048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9\" (UID: \"26324c17-9964-4c55-8f6c-30c6511f7048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:00.340753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.340734    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"]
Nov 03 11:15:00.340893 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.340873    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:00.347461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.347437    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"]
Nov 03 11:15:00.347584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.347562    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:00.353380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.353357    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:00.358758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.358735    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"]
Nov 03 11:15:00.363556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.363535    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"]
Nov 03 11:15:00.368034 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.368008    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"]
Nov 03 11:15:00.374030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.374006    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"]
Nov 03 11:15:00.379089 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.379069    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"]
Nov 03 11:15:00.379222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.379195    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:00.384029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.384009    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:00.386983 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.386904    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"]
Nov 03 11:15:00.391547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.391486    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"]
Nov 03 11:15:00.395583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.395565    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"]
Nov 03 11:15:00.400894 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.400876    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"]
Nov 03 11:15:00.401064 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.401044    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:00.405716 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.405697    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:00.407995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.407968    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"]
Nov 03 11:15:00.412053 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.412036    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"]
Nov 03 11:15:00.438071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438042    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp7n\" (UniqueName: \"kubernetes.io/projected/26324c17-9964-4c55-8f6c-30c6511f7048-kube-api-access-whp7n\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9\" (UID: \"26324c17-9964-4c55-8f6c-30c6511f7048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:00.438274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438087    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq\" (UID: \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:00.438274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438125    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdscx\" (UniqueName: \"kubernetes.io/projected/51df457b-9f93-446b-8f3a-2d4dd62cfe92-kube-api-access-rdscx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq\" (UID: \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:00.438274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438232    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1107f46f-c86c-4569-b755-9aa28e256af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n\" (UID: \"1107f46f-c86c-4569-b755-9aa28e256af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:00.438456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438272    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542lf\" (UniqueName: \"kubernetes.io/projected/2e754a45-81bc-45e1-9949-5c8c9f6a638c-kube-api-access-542lf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk\" (UID: \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:00.438456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438316    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf\" (UID: \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:00.438456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438364    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z76t9\" (UniqueName: \"kubernetes.io/projected/f191c99e-0c1a-4f16-9a57-231634a6cd6a-kube-api-access-z76t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd\" (UID: \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:00.438557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438483    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqn5\" (UniqueName: \"kubernetes.io/projected/b5cea438-8f15-4414-8a1c-7f6df195d5a7-kube-api-access-ctqn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj\" (UID: \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:00.438557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438544    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5\" (UID: \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:00.438692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438675    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xrx\" (UniqueName: \"kubernetes.io/projected/1107f46f-c86c-4569-b755-9aa28e256af0-kube-api-access-99xrx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n\" (UID: \"1107f46f-c86c-4569-b755-9aa28e256af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:00.438757 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438711    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e754a45-81bc-45e1-9949-5c8c9f6a638c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk\" (UID: \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:00.438757 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438746    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191c99e-0c1a-4f16-9a57-231634a6cd6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd\" (UID: \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:00.438871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438802    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40de722e-3cd0-4923-9aec-bc5695a58f9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b\" (UID: \"40de722e-3cd0-4923-9aec-bc5695a58f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:00.438871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438842    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6\" (UID: \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:00.439029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438889    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1107f46f-c86c-4569-b755-9aa28e256af0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n\" (UID: \"1107f46f-c86c-4569-b755-9aa28e256af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:00.439029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.438960    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt662\" (UniqueName: \"kubernetes.io/projected/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-kube-api-access-qt662\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5\" (UID: \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:00.439140 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439097    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88z7\" (UniqueName: \"kubernetes.io/projected/2534e7e5-ea82-432e-af50-cc00bb0fa611-kube-api-access-k88z7\") pod \"maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg\" (UID: \"2534e7e5-ea82-432e-af50-cc00bb0fa611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:00.439204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439158    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr\" (UID: \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:00.439307 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439223    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2534e7e5-ea82-432e-af50-cc00bb0fa611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg\" (UID: \"2534e7e5-ea82-432e-af50-cc00bb0fa611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:00.439307 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439273    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppwj\" (UniqueName: \"kubernetes.io/projected/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-kube-api-access-sppwj\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr\" (UID: \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:00.439307 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439302    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51df457b-9f93-446b-8f3a-2d4dd62cfe92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq\" (UID: \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:00.439530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439332    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-kube-api-access-m8qj6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf\" (UID: \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:00.439530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439370    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5\" (UID: \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:00.439530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439411    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5dgr\" (UniqueName: \"kubernetes.io/projected/36bf3819-7aea-4e1c-ad10-66f59268dcf1-kube-api-access-s5dgr\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl\" (UID: \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:00.439530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439434    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e754a45-81bc-45e1-9949-5c8c9f6a638c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk\" (UID: \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:00.439530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439474    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26324c17-9964-4c55-8f6c-30c6511f7048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9\" (UID: \"26324c17-9964-4c55-8f6c-30c6511f7048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:00.439530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439517    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cea438-8f15-4414-8a1c-7f6df195d5a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj\" (UID: \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:00.440006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439570    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5znx\" (UniqueName: \"kubernetes.io/projected/40de722e-3cd0-4923-9aec-bc5695a58f9e-kube-api-access-t5znx\") pod \"maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b\" (UID: \"40de722e-3cd0-4923-9aec-bc5695a58f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:00.440006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439574    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6\" (UID: \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:00.440006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439607    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf3819-7aea-4e1c-ad10-66f59268dcf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl\" (UID: \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:00.440006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439655    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdhq\" (UniqueName: \"kubernetes.io/projected/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-kube-api-access-9bdhq\") pod \"maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6\" (UID: \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:00.440006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439689    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6w7f\" (UniqueName: \"kubernetes.io/projected/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-kube-api-access-v6w7f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq\" (UID: \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:00.440006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.439882    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr\" (UID: \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:00.440366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.440344    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26324c17-9964-4c55-8f6c-30c6511f7048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9\" (UID: \"26324c17-9964-4c55-8f6c-30c6511f7048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:00.440425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.440348    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf3819-7aea-4e1c-ad10-66f59268dcf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl\" (UID: \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:00.449092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.449065    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdhq\" (UniqueName: \"kubernetes.io/projected/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-kube-api-access-9bdhq\") pod \"maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6\" (UID: \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:00.450278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.450251    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt662\" (UniqueName: \"kubernetes.io/projected/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-kube-api-access-qt662\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5\" (UID: \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:00.450975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.450933    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-542lf\" (UniqueName: \"kubernetes.io/projected/2e754a45-81bc-45e1-9949-5c8c9f6a638c-kube-api-access-542lf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk\" (UID: \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:00.452619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.452590    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xrx\" (UniqueName: \"kubernetes.io/projected/1107f46f-c86c-4569-b755-9aa28e256af0-kube-api-access-99xrx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n\" (UID: \"1107f46f-c86c-4569-b755-9aa28e256af0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:00.453539 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.453510    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppwj\" (UniqueName: \"kubernetes.io/projected/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-kube-api-access-sppwj\") pod \"maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr\" (UID: \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:00.456131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.456104    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5dgr\" (UniqueName: \"kubernetes.io/projected/36bf3819-7aea-4e1c-ad10-66f59268dcf1-kube-api-access-s5dgr\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl\" (UID: \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:00.459231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.459083    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp7n\" (UniqueName: \"kubernetes.io/projected/26324c17-9964-4c55-8f6c-30c6511f7048-kube-api-access-whp7n\") pod \"maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9\" (UID: \"26324c17-9964-4c55-8f6c-30c6511f7048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:00.476169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.476135    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:00.493569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.493545    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:00.542952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.542582    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51df457b-9f93-446b-8f3a-2d4dd62cfe92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq\" (UID: \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:00.542952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.542643    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-kube-api-access-m8qj6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf\" (UID: \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:00.542952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.542734    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cea438-8f15-4414-8a1c-7f6df195d5a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj\" (UID: \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:00.542952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.542782    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5znx\" (UniqueName: \"kubernetes.io/projected/40de722e-3cd0-4923-9aec-bc5695a58f9e-kube-api-access-t5znx\") pod \"maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b\" (UID: \"40de722e-3cd0-4923-9aec-bc5695a58f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:00.542952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.542837    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6w7f\" (UniqueName: \"kubernetes.io/projected/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-kube-api-access-v6w7f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq\" (UID: \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.543950    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cea438-8f15-4414-8a1c-7f6df195d5a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj\" (UID: \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.543995    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq\" (UID: \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.544050    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdscx\" (UniqueName: \"kubernetes.io/projected/51df457b-9f93-446b-8f3a-2d4dd62cfe92-kube-api-access-rdscx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq\" (UID: \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.544126    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf\" (UID: \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.544172    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z76t9\" (UniqueName: \"kubernetes.io/projected/f191c99e-0c1a-4f16-9a57-231634a6cd6a-kube-api-access-z76t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd\" (UID: \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.544203    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqn5\" (UniqueName: \"kubernetes.io/projected/b5cea438-8f15-4414-8a1c-7f6df195d5a7-kube-api-access-ctqn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj\" (UID: \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.544351    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191c99e-0c1a-4f16-9a57-231634a6cd6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd\" (UID: \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.544402    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40de722e-3cd0-4923-9aec-bc5695a58f9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b\" (UID: \"40de722e-3cd0-4923-9aec-bc5695a58f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.544461    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88z7\" (UniqueName: \"kubernetes.io/projected/2534e7e5-ea82-432e-af50-cc00bb0fa611-kube-api-access-k88z7\") pod \"maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg\" (UID: \"2534e7e5-ea82-432e-af50-cc00bb0fa611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.544528    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2534e7e5-ea82-432e-af50-cc00bb0fa611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg\" (UID: \"2534e7e5-ea82-432e-af50-cc00bb0fa611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.545215    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf\" (UID: \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.545613    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2534e7e5-ea82-432e-af50-cc00bb0fa611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg\" (UID: \"2534e7e5-ea82-432e-af50-cc00bb0fa611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.546420    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq\" (UID: \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.547385    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191c99e-0c1a-4f16-9a57-231634a6cd6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd\" (UID: \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.547515    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40de722e-3cd0-4923-9aec-bc5695a58f9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b\" (UID: \"40de722e-3cd0-4923-9aec-bc5695a58f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:00.548836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.547717    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:00.551706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.550232    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51df457b-9f93-446b-8f3a-2d4dd62cfe92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq\" (UID: \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:00.560279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.560251    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:00.561658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.561619    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqn5\" (UniqueName: \"kubernetes.io/projected/b5cea438-8f15-4414-8a1c-7f6df195d5a7-kube-api-access-ctqn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj\" (UID: \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:00.561995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.561966    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88z7\" (UniqueName: \"kubernetes.io/projected/2534e7e5-ea82-432e-af50-cc00bb0fa611-kube-api-access-k88z7\") pod \"maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg\" (UID: \"2534e7e5-ea82-432e-af50-cc00bb0fa611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:00.574571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.574541    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:00.586363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.586306    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6w7f\" (UniqueName: \"kubernetes.io/projected/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-kube-api-access-v6w7f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq\" (UID: \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:00.587673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.587609    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5znx\" (UniqueName: \"kubernetes.io/projected/40de722e-3cd0-4923-9aec-bc5695a58f9e-kube-api-access-t5znx\") pod \"maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b\" (UID: \"40de722e-3cd0-4923-9aec-bc5695a58f9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:00.588593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.588440    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z76t9\" (UniqueName: \"kubernetes.io/projected/f191c99e-0c1a-4f16-9a57-231634a6cd6a-kube-api-access-z76t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd\" (UID: \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:00.596581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.596550    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-kube-api-access-m8qj6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf\" (UID: \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:00.597523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.597468    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdscx\" (UniqueName: \"kubernetes.io/projected/51df457b-9f93-446b-8f3a-2d4dd62cfe92-kube-api-access-rdscx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq\" (UID: \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:00.607990 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:00.607956    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:15:00.627737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.627458    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:00.632514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.632412    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:00.662411 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.662366    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:00.676051 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.675864    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:00.681526 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.681325    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:00.696849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.696818    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:00.704653 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.704459    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:00.718559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.718537    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:00.727318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:00.727229    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:01.001364 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.001250    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"]
Nov 03 11:15:01.008558 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.008524    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"]
Nov 03 11:15:01.302681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.302642    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"]
Nov 03 11:15:01.307454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.307429    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"]
Nov 03 11:15:01.333031 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:01.332989    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1107f46f_c86c_4569_b755_9aa28e256af0.slice/crio-c0d5947e5bb29764f4c41d999302e39b8054deae577c11cee8a8ac48de5b754c WatchSource:0}: Error finding container c0d5947e5bb29764f4c41d999302e39b8054deae577c11cee8a8ac48de5b754c: Status 404 returned error can't find the container with id c0d5947e5bb29764f4c41d999302e39b8054deae577c11cee8a8ac48de5b754c
Nov 03 11:15:01.334763 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:01.334738    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bf3819_7aea_4e1c_ad10_66f59268dcf1.slice/crio-7f6f30b6d80a865390c2b89312c6b84de162697eb6f583f24a637454d727c4d6 WatchSource:0}: Error finding container 7f6f30b6d80a865390c2b89312c6b84de162697eb6f583f24a637454d727c4d6: Status 404 returned error can't find the container with id 7f6f30b6d80a865390c2b89312c6b84de162697eb6f583f24a637454d727c4d6
Nov 03 11:15:01.349058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.349032    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"]
Nov 03 11:15:01.352719 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:01.352685    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763c6d00_52d4_4cc5_af13_1e7cfa1fc700.slice/crio-3345c6c1a6ffe4c9086cea59601e8b0dfd4783547cfda4996fc294e7f2c412f9 WatchSource:0}: Error finding container 3345c6c1a6ffe4c9086cea59601e8b0dfd4783547cfda4996fc294e7f2c412f9: Status 404 returned error can't find the container with id 3345c6c1a6ffe4c9086cea59601e8b0dfd4783547cfda4996fc294e7f2c412f9
Nov 03 11:15:01.418384 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.418355    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"]
Nov 03 11:15:01.453764 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.453474    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"]
Nov 03 11:15:01.459118 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.459092    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"]
Nov 03 11:15:01.463700 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.463683    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"]
Nov 03 11:15:01.467769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.467745    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"]
Nov 03 11:15:01.491044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.491017    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"]
Nov 03 11:15:01.491876 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:01.491847    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bea47f_c0e1_4a14_acd0_920b5b43e4b3.slice/crio-02011c39cba6030fcda7165a3df5ac1d9a94e36006d691ec5fec70596ed774c3 WatchSource:0}: Error finding container 02011c39cba6030fcda7165a3df5ac1d9a94e36006d691ec5fec70596ed774c3: Status 404 returned error can't find the container with id 02011c39cba6030fcda7165a3df5ac1d9a94e36006d691ec5fec70596ed774c3
Nov 03 11:15:01.550770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.550733    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"]
Nov 03 11:15:01.555553 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.555526    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"]
Nov 03 11:15:01.562355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.562334    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"]
Nov 03 11:15:01.586245 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:01.586212    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51df457b_9f93_446b_8f3a_2d4dd62cfe92.slice/crio-abca26aaf7d72c9775af37a3ba56f8b5386f4db9ec78da6c650848aaa2b91a58 WatchSource:0}: Error finding container abca26aaf7d72c9775af37a3ba56f8b5386f4db9ec78da6c650848aaa2b91a58: Status 404 returned error can't find the container with id abca26aaf7d72c9775af37a3ba56f8b5386f4db9ec78da6c650848aaa2b91a58
Nov 03 11:15:01.661785 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.661682    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" event={"ID":"26324c17-9964-4c55-8f6c-30c6511f7048","Type":"ContainerStarted","Data":"b65d397f03f6b9ca48dfcf4131d92805a480217cf297707a6f6357f78f7bca98"}
Nov 03 11:15:01.663270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.663247    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" event={"ID":"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5","Type":"ContainerStarted","Data":"859fc22756d3094817ec39a97c30731dbc66ec6748bc9124408ba944bddf2cb5"}
Nov 03 11:15:01.664751 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.664717    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" event={"ID":"763c6d00-52d4-4cc5-af13-1e7cfa1fc700","Type":"ContainerStarted","Data":"3345c6c1a6ffe4c9086cea59601e8b0dfd4783547cfda4996fc294e7f2c412f9"}
Nov 03 11:15:01.666231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.666207    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" event={"ID":"1107f46f-c86c-4569-b755-9aa28e256af0","Type":"ContainerStarted","Data":"c0d5947e5bb29764f4c41d999302e39b8054deae577c11cee8a8ac48de5b754c"}
Nov 03 11:15:01.667662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.667639    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" event={"ID":"36bf3819-7aea-4e1c-ad10-66f59268dcf1","Type":"ContainerStarted","Data":"7f6f30b6d80a865390c2b89312c6b84de162697eb6f583f24a637454d727c4d6"}
Nov 03 11:15:01.669094 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.669072    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" event={"ID":"74bea47f-c0e1-4a14-acd0-920b5b43e4b3","Type":"ContainerStarted","Data":"02011c39cba6030fcda7165a3df5ac1d9a94e36006d691ec5fec70596ed774c3"}
Nov 03 11:15:01.670404 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:01.670382    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" event={"ID":"51df457b-9f93-446b-8f3a-2d4dd62cfe92","Type":"ContainerStarted","Data":"abca26aaf7d72c9775af37a3ba56f8b5386f4db9ec78da6c650848aaa2b91a58"}
Nov 03 11:15:01.690680 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:01.690648    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e754a45_81bc_45e1_9949_5c8c9f6a638c.slice/crio-ebdcaac6390e4449e26a087cfddbe491b1a25a824f434e3866602bb986d1358b WatchSource:0}: Error finding container ebdcaac6390e4449e26a087cfddbe491b1a25a824f434e3866602bb986d1358b: Status 404 returned error can't find the container with id ebdcaac6390e4449e26a087cfddbe491b1a25a824f434e3866602bb986d1358b
Nov 03 11:15:01.692099 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:01.691834    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40de722e_3cd0_4923_9aec_bc5695a58f9e.slice/crio-ae936f9dcfc936519a18d7128ede19c10d8b5f11e1ab94848ae514538fd8f932 WatchSource:0}: Error finding container ae936f9dcfc936519a18d7128ede19c10d8b5f11e1ab94848ae514538fd8f932: Status 404 returned error can't find the container with id ae936f9dcfc936519a18d7128ede19c10d8b5f11e1ab94848ae514538fd8f932
Nov 03 11:15:01.694279 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:01.694108    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb58eeb4_c1d2_449f_90d7_f92ab4eb1271.slice/crio-807fc7db27534b9b6a8bfbcda7459ea042aa85d7ff9a28ff913e507bff3856e9 WatchSource:0}: Error finding container 807fc7db27534b9b6a8bfbcda7459ea042aa85d7ff9a28ff913e507bff3856e9: Status 404 returned error can't find the container with id 807fc7db27534b9b6a8bfbcda7459ea042aa85d7ff9a28ff913e507bff3856e9
Nov 03 11:15:02.697261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.697227    5393 generic.go:356] "Generic (PLEG): container finished" podID="26324c17-9964-4c55-8f6c-30c6511f7048" containerID="7a6077e05383bac6851df5705e7a214ff50083f478054ac8f8e0f8628b7a73d9" exitCode=60
Nov 03 11:15:02.697731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.697317    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" event={"ID":"26324c17-9964-4c55-8f6c-30c6511f7048","Type":"ContainerDied","Data":"7a6077e05383bac6851df5705e7a214ff50083f478054ac8f8e0f8628b7a73d9"}
Nov 03 11:15:02.697813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.697795    5393 scope.go:122] "RemoveContainer" containerID="7a6077e05383bac6851df5705e7a214ff50083f478054ac8f8e0f8628b7a73d9"
Nov 03 11:15:02.700755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.700721    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" event={"ID":"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81","Type":"ContainerStarted","Data":"fe3393a39222d19fc68aa44d3b5a027b5d3230f7c67469ac3caea259536607fb"}
Nov 03 11:15:02.703340 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.703310    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" event={"ID":"f191c99e-0c1a-4f16-9a57-231634a6cd6a","Type":"ContainerStarted","Data":"c274957a128baaba39389cff9d4805315fb2273f08d34537aac88410aa94e180"}
Nov 03 11:15:02.703436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.703367    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" event={"ID":"f191c99e-0c1a-4f16-9a57-231634a6cd6a","Type":"ContainerStarted","Data":"5b0d78aa25b2988a00b73e890afd3aea1d71bc2f3bbd066a230822a673e9f5a5"}
Nov 03 11:15:02.707979 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.707906    5393 generic.go:356] "Generic (PLEG): container finished" podID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerID="f59124e20a27e4e85b59c65fa0b1748f1ed89fcc322ad71477358afcee552136" exitCode=60
Nov 03 11:15:02.708095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.708046    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" event={"ID":"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5","Type":"ContainerDied","Data":"f59124e20a27e4e85b59c65fa0b1748f1ed89fcc322ad71477358afcee552136"}
Nov 03 11:15:02.708356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.708340    5393 scope.go:122] "RemoveContainer" containerID="f59124e20a27e4e85b59c65fa0b1748f1ed89fcc322ad71477358afcee552136"
Nov 03 11:15:02.709871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.709849    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" event={"ID":"763c6d00-52d4-4cc5-af13-1e7cfa1fc700","Type":"ContainerStarted","Data":"c6d4cf8b5cecf8b9053a6da9984b257f067b44ec04853093c99385df5b34480f"}
Nov 03 11:15:02.711580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.711556    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" event={"ID":"1107f46f-c86c-4569-b755-9aa28e256af0","Type":"ContainerStarted","Data":"e0da9604cd4638344e9d78ce127b4df179e17102739055593ff1e63486a35091"}
Nov 03 11:15:02.714234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.714208    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" event={"ID":"36bf3819-7aea-4e1c-ad10-66f59268dcf1","Type":"ContainerStarted","Data":"01704c90f1884413983cd885290e7c385e38ac228957381770a0361ce4a4ecfd"}
Nov 03 11:15:02.716098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.716071    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" event={"ID":"b5cea438-8f15-4414-8a1c-7f6df195d5a7","Type":"ContainerStarted","Data":"12f1f39cb2a0ebf2b2bbde01229f6b96ba4fd030a3e6f593dcddb9041f44dbe0"}
Nov 03 11:15:02.716181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.716118    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" event={"ID":"b5cea438-8f15-4414-8a1c-7f6df195d5a7","Type":"ContainerStarted","Data":"dcbc23210c82fb5899f7232649bdf032cbfa64d25c5938966a89d49d501e3515"}
Nov 03 11:15:02.718289 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.718268    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" event={"ID":"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271","Type":"ContainerStarted","Data":"9c84217c525d1547ae4122666ab0af37fcd2e39d75a34540eb9daf006b44fc34"}
Nov 03 11:15:02.718378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.718305    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" event={"ID":"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271","Type":"ContainerStarted","Data":"807fc7db27534b9b6a8bfbcda7459ea042aa85d7ff9a28ff913e507bff3856e9"}
Nov 03 11:15:02.720046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.719990    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" event={"ID":"2e754a45-81bc-45e1-9949-5c8c9f6a638c","Type":"ContainerStarted","Data":"befaa0179cbe41ba735c03d377c3c513ca60aa19589388dc6757d3b72956b3c6"}
Nov 03 11:15:02.720153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.720071    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" event={"ID":"2e754a45-81bc-45e1-9949-5c8c9f6a638c","Type":"ContainerStarted","Data":"ebdcaac6390e4449e26a087cfddbe491b1a25a824f434e3866602bb986d1358b"}
Nov 03 11:15:02.721825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.721804    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" event={"ID":"51df457b-9f93-446b-8f3a-2d4dd62cfe92","Type":"ContainerStarted","Data":"9a7f5914d1ef8fbee9e9268ad24967d666a5897283e43a1fdf90a55e85e90710"}
Nov 03 11:15:02.723876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.723852    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" event={"ID":"2534e7e5-ea82-432e-af50-cc00bb0fa611","Type":"ContainerStarted","Data":"6e9f3b9805e8cc6fafafb469f59efce9f5a4cd70f11cc7b8d6761d6a03a008c8"}
Nov 03 11:15:02.724026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.723897    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" event={"ID":"2534e7e5-ea82-432e-af50-cc00bb0fa611","Type":"ContainerStarted","Data":"e3a9de48c839d5a23b800f22dff1d65f2f03951ab735a3c307e32ae288dd2ef7"}
Nov 03 11:15:02.725599 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.725580    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" event={"ID":"40de722e-3cd0-4923-9aec-bc5695a58f9e","Type":"ContainerStarted","Data":"e03161311c780187b9b59197b606f0ffc4edcd9cfd5771e3ac350e14900b01c2"}
Nov 03 11:15:02.725688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.725620    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" event={"ID":"40de722e-3cd0-4923-9aec-bc5695a58f9e","Type":"ContainerStarted","Data":"ae936f9dcfc936519a18d7128ede19c10d8b5f11e1ab94848ae514538fd8f932"}
Nov 03 11:15:02.747639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.747582    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" podStartSLOduration=2.747564787 podStartE2EDuration="2.747564787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.744373923 +0000 UTC m=+2746.649228365" watchObservedRunningTime="2026-11-03 11:15:02.747564787 +0000 UTC m=+2746.652419229"
Nov 03 11:15:02.759892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.759839    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" podStartSLOduration=2.75982571 podStartE2EDuration="2.75982571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.758558436 +0000 UTC m=+2746.663412878" watchObservedRunningTime="2026-11-03 11:15:02.75982571 +0000 UTC m=+2746.664680152"
Nov 03 11:15:02.774744 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.774645    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" podStartSLOduration=2.774630833 podStartE2EDuration="2.774630833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.773028482 +0000 UTC m=+2746.677882924" watchObservedRunningTime="2026-11-03 11:15:02.774630833 +0000 UTC m=+2746.679485273"
Nov 03 11:15:02.797109 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.796690    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" podStartSLOduration=2.796668865 podStartE2EDuration="2.796668865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.788394835 +0000 UTC m=+2746.693249276" watchObservedRunningTime="2026-11-03 11:15:02.796668865 +0000 UTC m=+2746.701523309"
Nov 03 11:15:02.808580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.808523    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" podStartSLOduration=2.808505819 podStartE2EDuration="2.808505819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.80611223 +0000 UTC m=+2746.710966671" watchObservedRunningTime="2026-11-03 11:15:02.808505819 +0000 UTC m=+2746.713360261"
Nov 03 11:15:02.839150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.839105    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" podStartSLOduration=2.839092159 podStartE2EDuration="2.839092159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.837069288 +0000 UTC m=+2746.741923725" watchObservedRunningTime="2026-11-03 11:15:02.839092159 +0000 UTC m=+2746.743946600"
Nov 03 11:15:02.870082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.870027    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" podStartSLOduration=2.870013542 podStartE2EDuration="2.870013542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.867015837 +0000 UTC m=+2746.771870277" watchObservedRunningTime="2026-11-03 11:15:02.870013542 +0000 UTC m=+2746.774867983"
Nov 03 11:15:02.896557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.896502    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" podStartSLOduration=2.896488112 podStartE2EDuration="2.896488112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.8911484 +0000 UTC m=+2746.796002841" watchObservedRunningTime="2026-11-03 11:15:02.896488112 +0000 UTC m=+2746.801342553"
Nov 03 11:15:02.921976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.921903    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" podStartSLOduration=2.92188848 podStartE2EDuration="2.92188848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.915817599 +0000 UTC m=+2746.820672039" watchObservedRunningTime="2026-11-03 11:15:02.92188848 +0000 UTC m=+2746.826742920"
Nov 03 11:15:02.959778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:02.959729    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" podStartSLOduration=2.959713522 podStartE2EDuration="2.959713522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:02.957592812 +0000 UTC m=+2746.862447255" watchObservedRunningTime="2026-11-03 11:15:02.959713522 +0000 UTC m=+2746.864567962"
Nov 03 11:15:03.736996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.736963    5393 generic.go:356] "Generic (PLEG): container finished" podID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerID="e03161311c780187b9b59197b606f0ffc4edcd9cfd5771e3ac350e14900b01c2" exitCode=60
Nov 03 11:15:03.737415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.737054    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" event={"ID":"40de722e-3cd0-4923-9aec-bc5695a58f9e","Type":"ContainerDied","Data":"e03161311c780187b9b59197b606f0ffc4edcd9cfd5771e3ac350e14900b01c2"}
Nov 03 11:15:03.737478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.737418    5393 scope.go:122] "RemoveContainer" containerID="e03161311c780187b9b59197b606f0ffc4edcd9cfd5771e3ac350e14900b01c2"
Nov 03 11:15:03.739531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.739499    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" event={"ID":"26324c17-9964-4c55-8f6c-30c6511f7048","Type":"ContainerStarted","Data":"e4481e3a89cd7b4d5eeee40b728356926a721dd8344435797e289f17c5afc77c"}
Nov 03 11:15:03.741450 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.741429    5393 generic.go:356] "Generic (PLEG): container finished" podID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerID="d15f76666ed37258f8b7ca26b1b05b0d1bc395eb135c519d7d6abe7e00f1cfa0" exitCode=60
Nov 03 11:15:03.741573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.741506    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" event={"ID":"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81","Type":"ContainerDied","Data":"d15f76666ed37258f8b7ca26b1b05b0d1bc395eb135c519d7d6abe7e00f1cfa0"}
Nov 03 11:15:03.741742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.741714    5393 scope.go:122] "RemoveContainer" containerID="d15f76666ed37258f8b7ca26b1b05b0d1bc395eb135c519d7d6abe7e00f1cfa0"
Nov 03 11:15:03.743534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.743508    5393 generic.go:356] "Generic (PLEG): container finished" podID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerID="c274957a128baaba39389cff9d4805315fb2273f08d34537aac88410aa94e180" exitCode=60
Nov 03 11:15:03.743672 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.743567    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" event={"ID":"f191c99e-0c1a-4f16-9a57-231634a6cd6a","Type":"ContainerDied","Data":"c274957a128baaba39389cff9d4805315fb2273f08d34537aac88410aa94e180"}
Nov 03 11:15:03.744298 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.744173    5393 scope.go:122] "RemoveContainer" containerID="c274957a128baaba39389cff9d4805315fb2273f08d34537aac88410aa94e180"
Nov 03 11:15:03.746961 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.746768    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" event={"ID":"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5","Type":"ContainerStarted","Data":"b9708256dcc93d3b303e7630b63c8b6b4a9cb765e9046971054197455d0a79c8"}
Nov 03 11:15:03.748862 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.748843    5393 generic.go:356] "Generic (PLEG): container finished" podID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerID="c6d4cf8b5cecf8b9053a6da9984b257f067b44ec04853093c99385df5b34480f" exitCode=60
Nov 03 11:15:03.749022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.748911    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" event={"ID":"763c6d00-52d4-4cc5-af13-1e7cfa1fc700","Type":"ContainerDied","Data":"c6d4cf8b5cecf8b9053a6da9984b257f067b44ec04853093c99385df5b34480f"}
Nov 03 11:15:03.749158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.749140    5393 scope.go:122] "RemoveContainer" containerID="c6d4cf8b5cecf8b9053a6da9984b257f067b44ec04853093c99385df5b34480f"
Nov 03 11:15:03.750773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.750749    5393 generic.go:356] "Generic (PLEG): container finished" podID="1107f46f-c86c-4569-b755-9aa28e256af0" containerID="e0da9604cd4638344e9d78ce127b4df179e17102739055593ff1e63486a35091" exitCode=60
Nov 03 11:15:03.750877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.750805    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" event={"ID":"1107f46f-c86c-4569-b755-9aa28e256af0","Type":"ContainerDied","Data":"e0da9604cd4638344e9d78ce127b4df179e17102739055593ff1e63486a35091"}
Nov 03 11:15:03.751057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.751041    5393 scope.go:122] "RemoveContainer" containerID="e0da9604cd4638344e9d78ce127b4df179e17102739055593ff1e63486a35091"
Nov 03 11:15:03.754968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.754945    5393 generic.go:356] "Generic (PLEG): container finished" podID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerID="01704c90f1884413983cd885290e7c385e38ac228957381770a0361ce4a4ecfd" exitCode=60
Nov 03 11:15:03.755070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.755036    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" event={"ID":"36bf3819-7aea-4e1c-ad10-66f59268dcf1","Type":"ContainerDied","Data":"01704c90f1884413983cd885290e7c385e38ac228957381770a0361ce4a4ecfd"}
Nov 03 11:15:03.755319 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.755300    5393 scope.go:122] "RemoveContainer" containerID="01704c90f1884413983cd885290e7c385e38ac228957381770a0361ce4a4ecfd"
Nov 03 11:15:03.757093 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.757015    5393 generic.go:356] "Generic (PLEG): container finished" podID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerID="12f1f39cb2a0ebf2b2bbde01229f6b96ba4fd030a3e6f593dcddb9041f44dbe0" exitCode=60
Nov 03 11:15:03.758311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.757117    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" event={"ID":"b5cea438-8f15-4414-8a1c-7f6df195d5a7","Type":"ContainerDied","Data":"12f1f39cb2a0ebf2b2bbde01229f6b96ba4fd030a3e6f593dcddb9041f44dbe0"}
Nov 03 11:15:03.758311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.757486    5393 scope.go:122] "RemoveContainer" containerID="12f1f39cb2a0ebf2b2bbde01229f6b96ba4fd030a3e6f593dcddb9041f44dbe0"
Nov 03 11:15:03.759563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.759542    5393 generic.go:356] "Generic (PLEG): container finished" podID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerID="9c84217c525d1547ae4122666ab0af37fcd2e39d75a34540eb9daf006b44fc34" exitCode=60
Nov 03 11:15:03.759656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.759621    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" event={"ID":"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271","Type":"ContainerDied","Data":"9c84217c525d1547ae4122666ab0af37fcd2e39d75a34540eb9daf006b44fc34"}
Nov 03 11:15:03.759835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.759820    5393 scope.go:122] "RemoveContainer" containerID="9c84217c525d1547ae4122666ab0af37fcd2e39d75a34540eb9daf006b44fc34"
Nov 03 11:15:03.761569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.761543    5393 generic.go:356] "Generic (PLEG): container finished" podID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerID="befaa0179cbe41ba735c03d377c3c513ca60aa19589388dc6757d3b72956b3c6" exitCode=60
Nov 03 11:15:03.761691 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.761623    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" event={"ID":"2e754a45-81bc-45e1-9949-5c8c9f6a638c","Type":"ContainerDied","Data":"befaa0179cbe41ba735c03d377c3c513ca60aa19589388dc6757d3b72956b3c6"}
Nov 03 11:15:03.761871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.761853    5393 scope.go:122] "RemoveContainer" containerID="befaa0179cbe41ba735c03d377c3c513ca60aa19589388dc6757d3b72956b3c6"
Nov 03 11:15:03.763533 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.763508    5393 generic.go:356] "Generic (PLEG): container finished" podID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerID="e55df5b9571ea4d4fae33698d7dde47099a9989d07ca75f761942c5bf241eb8c" exitCode=60
Nov 03 11:15:03.763640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.763583    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" event={"ID":"74bea47f-c0e1-4a14-acd0-920b5b43e4b3","Type":"ContainerDied","Data":"e55df5b9571ea4d4fae33698d7dde47099a9989d07ca75f761942c5bf241eb8c"}
Nov 03 11:15:03.764006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.763799    5393 scope.go:122] "RemoveContainer" containerID="e55df5b9571ea4d4fae33698d7dde47099a9989d07ca75f761942c5bf241eb8c"
Nov 03 11:15:03.770172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.770150    5393 generic.go:356] "Generic (PLEG): container finished" podID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerID="9a7f5914d1ef8fbee9e9268ad24967d666a5897283e43a1fdf90a55e85e90710" exitCode=60
Nov 03 11:15:03.770308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.770231    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" event={"ID":"51df457b-9f93-446b-8f3a-2d4dd62cfe92","Type":"ContainerDied","Data":"9a7f5914d1ef8fbee9e9268ad24967d666a5897283e43a1fdf90a55e85e90710"}
Nov 03 11:15:03.770492 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.770472    5393 scope.go:122] "RemoveContainer" containerID="9a7f5914d1ef8fbee9e9268ad24967d666a5897283e43a1fdf90a55e85e90710"
Nov 03 11:15:03.772446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.772421    5393 generic.go:356] "Generic (PLEG): container finished" podID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerID="6e9f3b9805e8cc6fafafb469f59efce9f5a4cd70f11cc7b8d6761d6a03a008c8" exitCode=60
Nov 03 11:15:03.772548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.772458    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" event={"ID":"2534e7e5-ea82-432e-af50-cc00bb0fa611","Type":"ContainerDied","Data":"6e9f3b9805e8cc6fafafb469f59efce9f5a4cd70f11cc7b8d6761d6a03a008c8"}
Nov 03 11:15:03.772801 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.772779    5393 scope.go:122] "RemoveContainer" containerID="6e9f3b9805e8cc6fafafb469f59efce9f5a4cd70f11cc7b8d6761d6a03a008c8"
Nov 03 11:15:03.852380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.852317    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" podStartSLOduration=3.852298529 podStartE2EDuration="3.852298529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:03.84785161 +0000 UTC m=+2747.752706051" watchObservedRunningTime="2026-11-03 11:15:03.852298529 +0000 UTC m=+2747.757152971"
Nov 03 11:15:03.916225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:03.915097    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" podStartSLOduration=3.9150822339999998 podStartE2EDuration="3.915082234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:03.912098159 +0000 UTC m=+2747.816952602" watchObservedRunningTime="2026-11-03 11:15:03.915082234 +0000 UTC m=+2747.819936674"
Nov 03 11:15:04.788143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.788044    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" event={"ID":"40de722e-3cd0-4923-9aec-bc5695a58f9e","Type":"ContainerStarted","Data":"c25ac1fbab87c2fa4acfee3a57e8f9765152d3a2d0687aaf115ada20c058e2fc"}
Nov 03 11:15:04.792024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.791988    5393 generic.go:356] "Generic (PLEG): container finished" podID="26324c17-9964-4c55-8f6c-30c6511f7048" containerID="e4481e3a89cd7b4d5eeee40b728356926a721dd8344435797e289f17c5afc77c" exitCode=60
Nov 03 11:15:04.792180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.792071    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" event={"ID":"26324c17-9964-4c55-8f6c-30c6511f7048","Type":"ContainerDied","Data":"e4481e3a89cd7b4d5eeee40b728356926a721dd8344435797e289f17c5afc77c"}
Nov 03 11:15:04.792180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.792138    5393 scope.go:122] "RemoveContainer" containerID="7a6077e05383bac6851df5705e7a214ff50083f478054ac8f8e0f8628b7a73d9"
Nov 03 11:15:04.792398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.792373    5393 scope.go:122] "RemoveContainer" containerID="e4481e3a89cd7b4d5eeee40b728356926a721dd8344435797e289f17c5afc77c"
Nov 03 11:15:04.792776 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:04.792753    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9_odh-ai-gateway-infra(26324c17-9964-4c55-8f6c-30c6511f7048)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" podUID="26324c17-9964-4c55-8f6c-30c6511f7048"
Nov 03 11:15:04.795118 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.795097    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" event={"ID":"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81","Type":"ContainerStarted","Data":"c91a65535eca8d272b7df491d20c6e9fec335525fb4e1fcec632673a7658b7ce"}
Nov 03 11:15:04.797446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.797419    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" event={"ID":"f191c99e-0c1a-4f16-9a57-231634a6cd6a","Type":"ContainerStarted","Data":"c299fd2bf7497a55316530a57ace332d9b36501243eea5113aa0d3cb9682d497"}
Nov 03 11:15:04.800718 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.800607    5393 generic.go:356] "Generic (PLEG): container finished" podID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerID="b9708256dcc93d3b303e7630b63c8b6b4a9cb765e9046971054197455d0a79c8" exitCode=60
Nov 03 11:15:04.800718 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.800707    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" event={"ID":"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5","Type":"ContainerDied","Data":"b9708256dcc93d3b303e7630b63c8b6b4a9cb765e9046971054197455d0a79c8"}
Nov 03 11:15:04.801249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.800992    5393 scope.go:122] "RemoveContainer" containerID="b9708256dcc93d3b303e7630b63c8b6b4a9cb765e9046971054197455d0a79c8"
Nov 03 11:15:04.801358 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:04.801338    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr_odh-ai-gateway-infra(1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5"
Nov 03 11:15:04.806098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.805958    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" event={"ID":"763c6d00-52d4-4cc5-af13-1e7cfa1fc700","Type":"ContainerStarted","Data":"db9680eda33a82d2bd7cafe381acb6dae800f217a6929879e8a0df883b3761f2"}
Nov 03 11:15:04.808401 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.808379    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" event={"ID":"1107f46f-c86c-4569-b755-9aa28e256af0","Type":"ContainerStarted","Data":"52b302ca7302c772ece0a02343f8046908ad2cdf2c3801566f56f937bc600bf9"}
Nov 03 11:15:04.810704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.810682    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" event={"ID":"36bf3819-7aea-4e1c-ad10-66f59268dcf1","Type":"ContainerStarted","Data":"7852ca8df5093500e739e754d6ec8a2567acc5482ffe808f8dcb9f923941c2e1"}
Nov 03 11:15:04.812770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.812750    5393 scope.go:122] "RemoveContainer" containerID="f59124e20a27e4e85b59c65fa0b1748f1ed89fcc322ad71477358afcee552136"
Nov 03 11:15:04.813094 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.813073    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" event={"ID":"b5cea438-8f15-4414-8a1c-7f6df195d5a7","Type":"ContainerStarted","Data":"04250294c4649d122fb0b3bdee781ccfab7e0fe31f9a37b24c2bb0daefc64a01"}
Nov 03 11:15:04.816445 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.816422    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" event={"ID":"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271","Type":"ContainerStarted","Data":"9ffdb64da60a48cd7c5d43939467dc3c3cbdcbff174993852ca54928ae7c0a3a"}
Nov 03 11:15:04.818614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.818588    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" event={"ID":"2e754a45-81bc-45e1-9949-5c8c9f6a638c","Type":"ContainerStarted","Data":"bd8618b8c6008f4d11054edd3cd81e0ef7d6c904ff28926dc09df695431abdfd"}
Nov 03 11:15:04.820611 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.820589    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" event={"ID":"74bea47f-c0e1-4a14-acd0-920b5b43e4b3","Type":"ContainerStarted","Data":"3ed3ac1d878b83c9d0d5dd06b25c55efa7e52a6997aeca708ec4b2be37477b37"}
Nov 03 11:15:04.822712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.822691    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" event={"ID":"51df457b-9f93-446b-8f3a-2d4dd62cfe92","Type":"ContainerStarted","Data":"c6236764a34e8e8be01cb06b7b9b8a0cae5ba527754fda885439190a66524816"}
Nov 03 11:15:04.830835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.830809    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" event={"ID":"2534e7e5-ea82-432e-af50-cc00bb0fa611","Type":"ContainerStarted","Data":"b2dc7e3dd1668dd0d58ccd46cf9e62786f237c95bf0484d9b9276e6d5a3500a6"}
Nov 03 11:15:04.876519 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:04.876467    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" podStartSLOduration=4.876452704 podStartE2EDuration="4.876452704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:04.872691098 +0000 UTC m=+2748.777545543" watchObservedRunningTime="2026-11-03 11:15:04.876452704 +0000 UTC m=+2748.781307145"
Nov 03 11:15:05.008813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.008761    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" podStartSLOduration=5.008748199 podStartE2EDuration="5.008748199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:05.00489502 +0000 UTC m=+2748.909749461" watchObservedRunningTime="2026-11-03 11:15:05.008748199 +0000 UTC m=+2748.913602640"
Nov 03 11:15:05.558969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.558932    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:15:05.884398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.884366    5393 generic.go:356] "Generic (PLEG): container finished" podID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerID="04250294c4649d122fb0b3bdee781ccfab7e0fe31f9a37b24c2bb0daefc64a01" exitCode=60
Nov 03 11:15:05.884902 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.884445    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" event={"ID":"b5cea438-8f15-4414-8a1c-7f6df195d5a7","Type":"ContainerDied","Data":"04250294c4649d122fb0b3bdee781ccfab7e0fe31f9a37b24c2bb0daefc64a01"}
Nov 03 11:15:05.884902 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.884516    5393 scope.go:122] "RemoveContainer" containerID="12f1f39cb2a0ebf2b2bbde01229f6b96ba4fd030a3e6f593dcddb9041f44dbe0"
Nov 03 11:15:05.885046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.885029    5393 scope.go:122] "RemoveContainer" containerID="04250294c4649d122fb0b3bdee781ccfab7e0fe31f9a37b24c2bb0daefc64a01"
Nov 03 11:15:05.885550 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.885525    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj_odh-ai-gateway-infra(b5cea438-8f15-4414-8a1c-7f6df195d5a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7"
Nov 03 11:15:05.887724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.887577    5393 generic.go:356] "Generic (PLEG): container finished" podID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerID="9ffdb64da60a48cd7c5d43939467dc3c3cbdcbff174993852ca54928ae7c0a3a" exitCode=60
Nov 03 11:15:05.887724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.887687    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" event={"ID":"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271","Type":"ContainerDied","Data":"9ffdb64da60a48cd7c5d43939467dc3c3cbdcbff174993852ca54928ae7c0a3a"}
Nov 03 11:15:05.888562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.888541    5393 scope.go:122] "RemoveContainer" containerID="9ffdb64da60a48cd7c5d43939467dc3c3cbdcbff174993852ca54928ae7c0a3a"
Nov 03 11:15:05.889130 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.889107    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf_odh-ai-gateway-infra(fb58eeb4-c1d2-449f-90d7-f92ab4eb1271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271"
Nov 03 11:15:05.891296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.891271    5393 generic.go:356] "Generic (PLEG): container finished" podID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerID="bd8618b8c6008f4d11054edd3cd81e0ef7d6c904ff28926dc09df695431abdfd" exitCode=60
Nov 03 11:15:05.891977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.891513    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" event={"ID":"2e754a45-81bc-45e1-9949-5c8c9f6a638c","Type":"ContainerDied","Data":"bd8618b8c6008f4d11054edd3cd81e0ef7d6c904ff28926dc09df695431abdfd"}
Nov 03 11:15:05.892077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.891976    5393 scope.go:122] "RemoveContainer" containerID="bd8618b8c6008f4d11054edd3cd81e0ef7d6c904ff28926dc09df695431abdfd"
Nov 03 11:15:05.892480 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.892444    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk_odh-ai-gateway-infra(2e754a45-81bc-45e1-9949-5c8c9f6a638c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c"
Nov 03 11:15:05.894059 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.894036    5393 generic.go:356] "Generic (PLEG): container finished" podID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerID="3ed3ac1d878b83c9d0d5dd06b25c55efa7e52a6997aeca708ec4b2be37477b37" exitCode=60
Nov 03 11:15:05.894180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.894165    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" event={"ID":"74bea47f-c0e1-4a14-acd0-920b5b43e4b3","Type":"ContainerDied","Data":"3ed3ac1d878b83c9d0d5dd06b25c55efa7e52a6997aeca708ec4b2be37477b37"}
Nov 03 11:15:05.895183 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.894856    5393 scope.go:122] "RemoveContainer" containerID="3ed3ac1d878b83c9d0d5dd06b25c55efa7e52a6997aeca708ec4b2be37477b37"
Nov 03 11:15:05.895613 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.895585    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5_odh-ai-gateway-infra(74bea47f-c0e1-4a14-acd0-920b5b43e4b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3"
Nov 03 11:15:05.897630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.897607    5393 generic.go:356] "Generic (PLEG): container finished" podID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerID="c6236764a34e8e8be01cb06b7b9b8a0cae5ba527754fda885439190a66524816" exitCode=60
Nov 03 11:15:05.897768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.897728    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" event={"ID":"51df457b-9f93-446b-8f3a-2d4dd62cfe92","Type":"ContainerDied","Data":"c6236764a34e8e8be01cb06b7b9b8a0cae5ba527754fda885439190a66524816"}
Nov 03 11:15:05.898136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.898119    5393 scope.go:122] "RemoveContainer" containerID="c6236764a34e8e8be01cb06b7b9b8a0cae5ba527754fda885439190a66524816"
Nov 03 11:15:05.899842 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.898498    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq_odh-ai-gateway-infra(51df457b-9f93-446b-8f3a-2d4dd62cfe92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92"
Nov 03 11:15:05.901967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.901710    5393 generic.go:356] "Generic (PLEG): container finished" podID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerID="b2dc7e3dd1668dd0d58ccd46cf9e62786f237c95bf0484d9b9276e6d5a3500a6" exitCode=60
Nov 03 11:15:05.901967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.901844    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" event={"ID":"2534e7e5-ea82-432e-af50-cc00bb0fa611","Type":"ContainerDied","Data":"b2dc7e3dd1668dd0d58ccd46cf9e62786f237c95bf0484d9b9276e6d5a3500a6"}
Nov 03 11:15:05.902503 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.902481    5393 scope.go:122] "RemoveContainer" containerID="b2dc7e3dd1668dd0d58ccd46cf9e62786f237c95bf0484d9b9276e6d5a3500a6"
Nov 03 11:15:05.902896 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.902871    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg_odh-ai-gateway-infra(2534e7e5-ea82-432e-af50-cc00bb0fa611)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611"
Nov 03 11:15:05.907859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.907834    5393 generic.go:356] "Generic (PLEG): container finished" podID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerID="c25ac1fbab87c2fa4acfee3a57e8f9765152d3a2d0687aaf115ada20c058e2fc" exitCode=60
Nov 03 11:15:05.908012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.907861    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" event={"ID":"40de722e-3cd0-4923-9aec-bc5695a58f9e","Type":"ContainerDied","Data":"c25ac1fbab87c2fa4acfee3a57e8f9765152d3a2d0687aaf115ada20c058e2fc"}
Nov 03 11:15:05.908221 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.908204    5393 scope.go:122] "RemoveContainer" containerID="c25ac1fbab87c2fa4acfee3a57e8f9765152d3a2d0687aaf115ada20c058e2fc"
Nov 03 11:15:05.908496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.908480    5393 scope.go:122] "RemoveContainer" containerID="9c84217c525d1547ae4122666ab0af37fcd2e39d75a34540eb9daf006b44fc34"
Nov 03 11:15:05.908570 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.908506    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b_odh-ai-gateway-infra(40de722e-3cd0-4923-9aec-bc5695a58f9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e"
Nov 03 11:15:05.910724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.910702    5393 scope.go:122] "RemoveContainer" containerID="e4481e3a89cd7b4d5eeee40b728356926a721dd8344435797e289f17c5afc77c"
Nov 03 11:15:05.911628 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.911098    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9_odh-ai-gateway-infra(26324c17-9964-4c55-8f6c-30c6511f7048)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" podUID="26324c17-9964-4c55-8f6c-30c6511f7048"
Nov 03 11:15:05.914763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.914738    5393 generic.go:356] "Generic (PLEG): container finished" podID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerID="c91a65535eca8d272b7df491d20c6e9fec335525fb4e1fcec632673a7658b7ce" exitCode=60
Nov 03 11:15:05.914879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.914841    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" event={"ID":"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81","Type":"ContainerDied","Data":"c91a65535eca8d272b7df491d20c6e9fec335525fb4e1fcec632673a7658b7ce"}
Nov 03 11:15:05.915288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.915265    5393 scope.go:122] "RemoveContainer" containerID="c91a65535eca8d272b7df491d20c6e9fec335525fb4e1fcec632673a7658b7ce"
Nov 03 11:15:05.915661 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.915637    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq_odh-ai-gateway-infra(4ccb486b-de82-43bc-b8fa-47e5a6a2bf81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81"
Nov 03 11:15:05.918161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.918130    5393 generic.go:356] "Generic (PLEG): container finished" podID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerID="c299fd2bf7497a55316530a57ace332d9b36501243eea5113aa0d3cb9682d497" exitCode=60
Nov 03 11:15:05.918281 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.918255    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" event={"ID":"f191c99e-0c1a-4f16-9a57-231634a6cd6a","Type":"ContainerDied","Data":"c299fd2bf7497a55316530a57ace332d9b36501243eea5113aa0d3cb9682d497"}
Nov 03 11:15:05.918604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.918583    5393 scope.go:122] "RemoveContainer" containerID="c299fd2bf7497a55316530a57ace332d9b36501243eea5113aa0d3cb9682d497"
Nov 03 11:15:05.919106 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.919078    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd_odh-ai-gateway-infra(f191c99e-0c1a-4f16-9a57-231634a6cd6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a"
Nov 03 11:15:05.920548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.920532    5393 scope.go:122] "RemoveContainer" containerID="b9708256dcc93d3b303e7630b63c8b6b4a9cb765e9046971054197455d0a79c8"
Nov 03 11:15:05.920986 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.920962    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr_odh-ai-gateway-infra(1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5"
Nov 03 11:15:05.926022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.925994    5393 generic.go:356] "Generic (PLEG): container finished" podID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerID="db9680eda33a82d2bd7cafe381acb6dae800f217a6929879e8a0df883b3761f2" exitCode=60
Nov 03 11:15:05.926138 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.926038    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" event={"ID":"763c6d00-52d4-4cc5-af13-1e7cfa1fc700","Type":"ContainerDied","Data":"db9680eda33a82d2bd7cafe381acb6dae800f217a6929879e8a0df883b3761f2"}
Nov 03 11:15:05.926373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.926351    5393 scope.go:122] "RemoveContainer" containerID="db9680eda33a82d2bd7cafe381acb6dae800f217a6929879e8a0df883b3761f2"
Nov 03 11:15:05.926700 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.926675    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6_odh-ai-gateway-infra(763c6d00-52d4-4cc5-af13-1e7cfa1fc700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700"
Nov 03 11:15:05.928193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.928174    5393 generic.go:356] "Generic (PLEG): container finished" podID="1107f46f-c86c-4569-b755-9aa28e256af0" containerID="52b302ca7302c772ece0a02343f8046908ad2cdf2c3801566f56f937bc600bf9" exitCode=60
Nov 03 11:15:05.928297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.928243    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" event={"ID":"1107f46f-c86c-4569-b755-9aa28e256af0","Type":"ContainerDied","Data":"52b302ca7302c772ece0a02343f8046908ad2cdf2c3801566f56f937bc600bf9"}
Nov 03 11:15:05.928535 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.928516    5393 scope.go:122] "RemoveContainer" containerID="52b302ca7302c772ece0a02343f8046908ad2cdf2c3801566f56f937bc600bf9"
Nov 03 11:15:05.928852 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.928830    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n_odh-ai-gateway-infra(1107f46f-c86c-4569-b755-9aa28e256af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" podUID="1107f46f-c86c-4569-b755-9aa28e256af0"
Nov 03 11:15:05.930523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.930502    5393 generic.go:356] "Generic (PLEG): container finished" podID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerID="7852ca8df5093500e739e754d6ec8a2567acc5482ffe808f8dcb9f923941c2e1" exitCode=60
Nov 03 11:15:05.930625 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.930537    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" event={"ID":"36bf3819-7aea-4e1c-ad10-66f59268dcf1","Type":"ContainerDied","Data":"7852ca8df5093500e739e754d6ec8a2567acc5482ffe808f8dcb9f923941c2e1"}
Nov 03 11:15:05.930804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.930789    5393 scope.go:122] "RemoveContainer" containerID="7852ca8df5093500e739e754d6ec8a2567acc5482ffe808f8dcb9f923941c2e1"
Nov 03 11:15:05.931114 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:05.931095    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl_odh-ai-gateway-infra(36bf3819-7aea-4e1c-ad10-66f59268dcf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1"
Nov 03 11:15:05.941590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.941564    5393 scope.go:122] "RemoveContainer" containerID="befaa0179cbe41ba735c03d377c3c513ca60aa19589388dc6757d3b72956b3c6"
Nov 03 11:15:05.960007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.959898    5393 scope.go:122] "RemoveContainer" containerID="e55df5b9571ea4d4fae33698d7dde47099a9989d07ca75f761942c5bf241eb8c"
Nov 03 11:15:05.977155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.977125    5393 scope.go:122] "RemoveContainer" containerID="9a7f5914d1ef8fbee9e9268ad24967d666a5897283e43a1fdf90a55e85e90710"
Nov 03 11:15:05.995547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:05.995521    5393 scope.go:122] "RemoveContainer" containerID="6e9f3b9805e8cc6fafafb469f59efce9f5a4cd70f11cc7b8d6761d6a03a008c8"
Nov 03 11:15:06.012126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.012103    5393 scope.go:122] "RemoveContainer" containerID="e03161311c780187b9b59197b606f0ffc4edcd9cfd5771e3ac350e14900b01c2"
Nov 03 11:15:06.029145 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.029124    5393 scope.go:122] "RemoveContainer" containerID="d15f76666ed37258f8b7ca26b1b05b0d1bc395eb135c519d7d6abe7e00f1cfa0"
Nov 03 11:15:06.047670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.047641    5393 scope.go:122] "RemoveContainer" containerID="c274957a128baaba39389cff9d4805315fb2273f08d34537aac88410aa94e180"
Nov 03 11:15:06.063957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.063911    5393 scope.go:122] "RemoveContainer" containerID="c6d4cf8b5cecf8b9053a6da9984b257f067b44ec04853093c99385df5b34480f"
Nov 03 11:15:06.085198 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.085177    5393 scope.go:122] "RemoveContainer" containerID="e0da9604cd4638344e9d78ce127b4df179e17102739055593ff1e63486a35091"
Nov 03 11:15:06.102283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.102254    5393 scope.go:122] "RemoveContainer" containerID="01704c90f1884413983cd885290e7c385e38ac228957381770a0361ce4a4ecfd"
Nov 03 11:15:06.942515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.942488    5393 scope.go:122] "RemoveContainer" containerID="c91a65535eca8d272b7df491d20c6e9fec335525fb4e1fcec632673a7658b7ce"
Nov 03 11:15:06.943037 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.943009    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq_odh-ai-gateway-infra(4ccb486b-de82-43bc-b8fa-47e5a6a2bf81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81"
Nov 03 11:15:06.944256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.944235    5393 scope.go:122] "RemoveContainer" containerID="c299fd2bf7497a55316530a57ace332d9b36501243eea5113aa0d3cb9682d497"
Nov 03 11:15:06.944615 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.944591    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd_odh-ai-gateway-infra(f191c99e-0c1a-4f16-9a57-231634a6cd6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a"
Nov 03 11:15:06.945782 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.945757    5393 scope.go:122] "RemoveContainer" containerID="db9680eda33a82d2bd7cafe381acb6dae800f217a6929879e8a0df883b3761f2"
Nov 03 11:15:06.946166 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.946140    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6_odh-ai-gateway-infra(763c6d00-52d4-4cc5-af13-1e7cfa1fc700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700"
Nov 03 11:15:06.947670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.947651    5393 scope.go:122] "RemoveContainer" containerID="52b302ca7302c772ece0a02343f8046908ad2cdf2c3801566f56f937bc600bf9"
Nov 03 11:15:06.947983 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.947961    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n_odh-ai-gateway-infra(1107f46f-c86c-4569-b755-9aa28e256af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" podUID="1107f46f-c86c-4569-b755-9aa28e256af0"
Nov 03 11:15:06.949322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.949302    5393 scope.go:122] "RemoveContainer" containerID="7852ca8df5093500e739e754d6ec8a2567acc5482ffe808f8dcb9f923941c2e1"
Nov 03 11:15:06.949590 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.949568    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl_odh-ai-gateway-infra(36bf3819-7aea-4e1c-ad10-66f59268dcf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1"
Nov 03 11:15:06.950872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.950855    5393 scope.go:122] "RemoveContainer" containerID="04250294c4649d122fb0b3bdee781ccfab7e0fe31f9a37b24c2bb0daefc64a01"
Nov 03 11:15:06.951155 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.951133    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj_odh-ai-gateway-infra(b5cea438-8f15-4414-8a1c-7f6df195d5a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7"
Nov 03 11:15:06.952511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.952491    5393 scope.go:122] "RemoveContainer" containerID="9ffdb64da60a48cd7c5d43939467dc3c3cbdcbff174993852ca54928ae7c0a3a"
Nov 03 11:15:06.952757 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.952734    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf_odh-ai-gateway-infra(fb58eeb4-c1d2-449f-90d7-f92ab4eb1271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271"
Nov 03 11:15:06.954276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.954258    5393 scope.go:122] "RemoveContainer" containerID="bd8618b8c6008f4d11054edd3cd81e0ef7d6c904ff28926dc09df695431abdfd"
Nov 03 11:15:06.954519 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.954498    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk_odh-ai-gateway-infra(2e754a45-81bc-45e1-9949-5c8c9f6a638c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c"
Nov 03 11:15:06.964361 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.964325    5393 scope.go:122] "RemoveContainer" containerID="3ed3ac1d878b83c9d0d5dd06b25c55efa7e52a6997aeca708ec4b2be37477b37"
Nov 03 11:15:06.965060 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.964911    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5_odh-ai-gateway-infra(74bea47f-c0e1-4a14-acd0-920b5b43e4b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3"
Nov 03 11:15:06.973598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.973558    5393 scope.go:122] "RemoveContainer" containerID="c6236764a34e8e8be01cb06b7b9b8a0cae5ba527754fda885439190a66524816"
Nov 03 11:15:06.974120 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.974089    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq_odh-ai-gateway-infra(51df457b-9f93-446b-8f3a-2d4dd62cfe92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92"
Nov 03 11:15:06.975939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.975904    5393 scope.go:122] "RemoveContainer" containerID="b2dc7e3dd1668dd0d58ccd46cf9e62786f237c95bf0484d9b9276e6d5a3500a6"
Nov 03 11:15:06.976444 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.976420    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg_odh-ai-gateway-infra(2534e7e5-ea82-432e-af50-cc00bb0fa611)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611"
Nov 03 11:15:06.978591 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:06.978576    5393 scope.go:122] "RemoveContainer" containerID="c25ac1fbab87c2fa4acfee3a57e8f9765152d3a2d0687aaf115ada20c058e2fc"
Nov 03 11:15:06.978845 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:06.978823    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b_odh-ai-gateway-infra(40de722e-3cd0-4923-9aec-bc5695a58f9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e"
Nov 03 11:15:07.572521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.570722    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"]
Nov 03 11:15:07.616511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.616476    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"]
Nov 03 11:15:07.616680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.616612    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.621935 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.621895    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9265bf-openshift-default-dockercfg-92kvj\""
Nov 03 11:15:07.731318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.731282    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.731502 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.731434    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-data\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.731550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.731531    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wnpf\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-kube-api-access-9wnpf\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.731606 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.731593    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-certs\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.731691 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.731675    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-istio-token\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.731761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.731734    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f889894-693a-4511-b839-d620a739a2c8-istio-podinfo\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.731854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.731841    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f889894-693a-4511-b839-d620a739a2c8-istiod-ca-cert\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.731991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.731976    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-envoy\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.732041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.732006    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-credential-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834241 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834153    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f889894-693a-4511-b839-d620a739a2c8-istiod-ca-cert\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834241 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834226    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-envoy\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834252    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-credential-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834307    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834380    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-data\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834453    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wnpf\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-kube-api-access-9wnpf\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834519    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-certs\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834569    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-istio-token\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834609    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f889894-693a-4511-b839-d620a739a2c8-istio-podinfo\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834655    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-credential-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834747    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-data\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834820    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-socket\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.834960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.834897    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f889894-693a-4511-b839-d620a739a2c8-istiod-ca-cert\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.835292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.835268    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-certs\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.836831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.836809    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-envoy\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.837650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.837633    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f889894-693a-4511-b839-d620a739a2c8-istio-podinfo\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.842995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.842970    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-istio-token\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.843662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.843643    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wnpf\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-kube-api-access-9wnpf\") pod \"e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") " pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:07.936437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:07.936405    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:08.232250 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:08.232223    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f889894_693a_4511_b839_d620a739a2c8.slice/crio-26c2458b6237b8b3182e9670bb7bd6afec2b697545c3e38d35b306b7e9c6724f WatchSource:0}: Error finding container 26c2458b6237b8b3182e9670bb7bd6afec2b697545c3e38d35b306b7e9c6724f: Status 404 returned error can't find the container with id 26c2458b6237b8b3182e9670bb7bd6afec2b697545c3e38d35b306b7e9c6724f
Nov 03 11:15:08.235835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:08.235796    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:08.235951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:08.235880    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:08.235951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:08.235944    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:08.236202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:08.236184    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"]
Nov 03 11:15:09.027343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:09.027304    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" event={"ID":"4f889894-693a-4511-b839-d620a739a2c8","Type":"ContainerStarted","Data":"2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d"}
Nov 03 11:15:09.027560 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:09.027357    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" event={"ID":"4f889894-693a-4511-b839-d620a739a2c8","Type":"ContainerStarted","Data":"26c2458b6237b8b3182e9670bb7bd6afec2b697545c3e38d35b306b7e9c6724f"}
Nov 03 11:15:09.051889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:09.051833    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podStartSLOduration=2.051818234 podStartE2EDuration="2.051818234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:09.046442659 +0000 UTC m=+2752.951297100" watchObservedRunningTime="2026-11-03 11:15:09.051818234 +0000 UTC m=+2752.956672744"
Nov 03 11:15:09.936549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:09.936523    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:09.938060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:09.938033    5393 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 11:15:09.938205 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:09.938081    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 11:15:10.737902 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:10.737868    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:15:10.937356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:10.937323    5393 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 11:15:10.937793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:10.937373    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 11:15:11.937371 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:11.937335    5393 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 11:15:11.937838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:11.937399    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 11:15:12.936985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:12.936905    5393 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 11:15:12.937205 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:12.937006    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 11:15:13.227448 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:13.227362    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:15:13.937947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:13.936949    5393 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 11:15:13.937947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:13.937023    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 11:15:14.414946 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:14.407494    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcfb069_7210_45e2_a830_5b4294e0ab20.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 11:15:14.944458 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:14.944419    5393 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 11:15:14.944785 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:14.944751    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 11:15:15.223077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:15.222984    5393 scope.go:122] "RemoveContainer" containerID="b9708256dcc93d3b303e7630b63c8b6b4a9cb765e9046971054197455d0a79c8"
Nov 03 11:15:15.942767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:15.942740    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:15.943244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:15.943152    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:15.943986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:15.943970    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:15:16.136552 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:16.136504    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" event={"ID":"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5","Type":"ContainerStarted","Data":"c82aa6c6f3c211ca8977a97e43148cb5904dc2f3945b99251f70b2e48d748e28"}
Nov 03 11:15:16.234700 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:16.234619    5393 scope.go:122] "RemoveContainer" containerID="c299fd2bf7497a55316530a57ace332d9b36501243eea5113aa0d3cb9682d497"
Nov 03 11:15:16.235187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:16.235159    5393 scope.go:122] "RemoveContainer" containerID="e4481e3a89cd7b4d5eeee40b728356926a721dd8344435797e289f17c5afc77c"
Nov 03 11:15:17.147409 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.147362    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" event={"ID":"26324c17-9964-4c55-8f6c-30c6511f7048","Type":"ContainerStarted","Data":"d89f53249ac243c78443963b697a1818dc70a2ed6db9ea371029813f8de9bbac"}
Nov 03 11:15:17.149593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.149568    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" event={"ID":"f191c99e-0c1a-4f16-9a57-231634a6cd6a","Type":"ContainerStarted","Data":"c9514698b25c42cedf5f09e26bfb7ee0ffddfb94891933dd1e733ae9b5c22f56"}
Nov 03 11:15:17.151522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.151500    5393 generic.go:356] "Generic (PLEG): container finished" podID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerID="c82aa6c6f3c211ca8977a97e43148cb5904dc2f3945b99251f70b2e48d748e28" exitCode=60
Nov 03 11:15:17.151609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.151519    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" event={"ID":"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5","Type":"ContainerDied","Data":"c82aa6c6f3c211ca8977a97e43148cb5904dc2f3945b99251f70b2e48d748e28"}
Nov 03 11:15:17.151609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.151585    5393 scope.go:122] "RemoveContainer" containerID="b9708256dcc93d3b303e7630b63c8b6b4a9cb765e9046971054197455d0a79c8"
Nov 03 11:15:17.151952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.151897    5393 scope.go:122] "RemoveContainer" containerID="c82aa6c6f3c211ca8977a97e43148cb5904dc2f3945b99251f70b2e48d748e28"
Nov 03 11:15:17.152322 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:17.152261    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr_odh-ai-gateway-infra(1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5"
Nov 03 11:15:17.175573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.175545    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"]
Nov 03 11:15:17.223300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.223271    5393 scope.go:122] "RemoveContainer" containerID="bd8618b8c6008f4d11054edd3cd81e0ef7d6c904ff28926dc09df695431abdfd"
Nov 03 11:15:17.471162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.471081    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"]
Nov 03 11:15:17.478006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.477981    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:17.484694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.484662    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"]
Nov 03 11:15:17.487525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.487480    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3eb43f\""
Nov 03 11:15:17.504133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.504097    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"]
Nov 03 11:15:17.517132 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.515902    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:17.530960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.526862    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3eb43f-dockercfg-hs4bw\""
Nov 03 11:15:17.530960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.529300    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"]
Nov 03 11:15:17.556284 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.556248    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d86bb9-ee25-410c-99d3-adc646178897-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4\" (UID: \"e3d86bb9-ee25-410c-99d3-adc646178897\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:17.556425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.556312    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxqp\" (UniqueName: \"kubernetes.io/projected/e3d86bb9-ee25-410c-99d3-adc646178897-kube-api-access-7lxqp\") pod \"payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4\" (UID: \"e3d86bb9-ee25-410c-99d3-adc646178897\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:17.725997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.710548    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d86bb9-ee25-410c-99d3-adc646178897-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4\" (UID: \"e3d86bb9-ee25-410c-99d3-adc646178897\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:17.725997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.710614    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxqp\" (UniqueName: \"kubernetes.io/projected/e3d86bb9-ee25-410c-99d3-adc646178897-kube-api-access-7lxqp\") pod \"payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4\" (UID: \"e3d86bb9-ee25-410c-99d3-adc646178897\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:17.725997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.710731    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxfcw\" (UniqueName: \"kubernetes.io/projected/b8cac8cf-3549-470b-b998-5cf85a1343bc-kube-api-access-vxfcw\") pod \"payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6\" (UID: \"b8cac8cf-3549-470b-b998-5cf85a1343bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:17.725997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.710765    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cac8cf-3549-470b-b998-5cf85a1343bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6\" (UID: \"b8cac8cf-3549-470b-b998-5cf85a1343bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:17.725997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.711897    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d86bb9-ee25-410c-99d3-adc646178897-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4\" (UID: \"e3d86bb9-ee25-410c-99d3-adc646178897\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:17.747945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.746014    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxqp\" (UniqueName: \"kubernetes.io/projected/e3d86bb9-ee25-410c-99d3-adc646178897-kube-api-access-7lxqp\") pod \"payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4\" (UID: \"e3d86bb9-ee25-410c-99d3-adc646178897\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:17.816946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.816823    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxfcw\" (UniqueName: \"kubernetes.io/projected/b8cac8cf-3549-470b-b998-5cf85a1343bc-kube-api-access-vxfcw\") pod \"payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6\" (UID: \"b8cac8cf-3549-470b-b998-5cf85a1343bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:17.816946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.816875    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cac8cf-3549-470b-b998-5cf85a1343bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6\" (UID: \"b8cac8cf-3549-470b-b998-5cf85a1343bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:17.825943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.821043    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cac8cf-3549-470b-b998-5cf85a1343bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6\" (UID: \"b8cac8cf-3549-470b-b998-5cf85a1343bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:17.845942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:17.843150    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxfcw\" (UniqueName: \"kubernetes.io/projected/b8cac8cf-3549-470b-b998-5cf85a1343bc-kube-api-access-vxfcw\") pod \"payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6\" (UID: \"b8cac8cf-3549-470b-b998-5cf85a1343bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:18.033980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.033887    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:18.062646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.062621    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:18.198990 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.191773    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"]
Nov 03 11:15:18.232960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.223787    5393 scope.go:122] "RemoveContainer" containerID="52b302ca7302c772ece0a02343f8046908ad2cdf2c3801566f56f937bc600bf9"
Nov 03 11:15:18.232960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.225566    5393 scope.go:122] "RemoveContainer" containerID="9ffdb64da60a48cd7c5d43939467dc3c3cbdcbff174993852ca54928ae7c0a3a"
Nov 03 11:15:18.232960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.227658    5393 scope.go:122] "RemoveContainer" containerID="7852ca8df5093500e739e754d6ec8a2567acc5482ffe808f8dcb9f923941c2e1"
Nov 03 11:15:18.232960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.228108    5393 scope.go:122] "RemoveContainer" containerID="3ed3ac1d878b83c9d0d5dd06b25c55efa7e52a6997aeca708ec4b2be37477b37"
Nov 03 11:15:18.235414 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.234375    5393 scope.go:122] "RemoveContainer" containerID="04250294c4649d122fb0b3bdee781ccfab7e0fe31f9a37b24c2bb0daefc64a01"
Nov 03 11:15:18.235414 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.235030    5393 scope.go:122] "RemoveContainer" containerID="b2dc7e3dd1668dd0d58ccd46cf9e62786f237c95bf0484d9b9276e6d5a3500a6"
Nov 03 11:15:18.235686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.235663    5393 scope.go:122] "RemoveContainer" containerID="db9680eda33a82d2bd7cafe381acb6dae800f217a6929879e8a0df883b3761f2"
Nov 03 11:15:18.236985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.236210    5393 scope.go:122] "RemoveContainer" containerID="c6236764a34e8e8be01cb06b7b9b8a0cae5ba527754fda885439190a66524816"
Nov 03 11:15:18.269177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.268648    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" event={"ID":"2e754a45-81bc-45e1-9949-5c8c9f6a638c","Type":"ContainerStarted","Data":"b092211f9d7c52071a71da72546d505506e3547fde9bc7aeffd7b23872305142"}
Nov 03 11:15:18.269177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.268708    5393 generic.go:356] "Generic (PLEG): container finished" podID="26324c17-9964-4c55-8f6c-30c6511f7048" containerID="d89f53249ac243c78443963b697a1818dc70a2ed6db9ea371029813f8de9bbac" exitCode=60
Nov 03 11:15:18.269177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.268724    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" event={"ID":"26324c17-9964-4c55-8f6c-30c6511f7048","Type":"ContainerDied","Data":"d89f53249ac243c78443963b697a1818dc70a2ed6db9ea371029813f8de9bbac"}
Nov 03 11:15:18.269177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.268761    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"]
Nov 03 11:15:18.269177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.269028    5393 scope.go:122] "RemoveContainer" containerID="e4481e3a89cd7b4d5eeee40b728356926a721dd8344435797e289f17c5afc77c"
Nov 03 11:15:18.277372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.277328    5393 generic.go:356] "Generic (PLEG): container finished" podID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerID="c9514698b25c42cedf5f09e26bfb7ee0ffddfb94891933dd1e733ae9b5c22f56" exitCode=60
Nov 03 11:15:18.277573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.277553    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" event={"ID":"f191c99e-0c1a-4f16-9a57-231634a6cd6a","Type":"ContainerDied","Data":"c9514698b25c42cedf5f09e26bfb7ee0ffddfb94891933dd1e733ae9b5c22f56"}
Nov 03 11:15:18.420344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.420209    5393 scope.go:122] "RemoveContainer" containerID="c299fd2bf7497a55316530a57ace332d9b36501243eea5113aa0d3cb9682d497"
Nov 03 11:15:18.598666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.598640    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"]
Nov 03 11:15:18.603086 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:18.603062    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d86bb9_ee25_410c_99d3_adc646178897.slice/crio-066e8560ff324e239dfeb42c0572d0939481eeafe19d82db75695ca08ec64240 WatchSource:0}: Error finding container 066e8560ff324e239dfeb42c0572d0939481eeafe19d82db75695ca08ec64240: Status 404 returned error can't find the container with id 066e8560ff324e239dfeb42c0572d0939481eeafe19d82db75695ca08ec64240
Nov 03 11:15:18.642140 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.642105    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"]
Nov 03 11:15:18.872162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.872136    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:18.874578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.874547    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26324c17-9964-4c55-8f6c-30c6511f7048-kube-api-access-whp7n\" (UniqueName: \"kubernetes.io/projected/26324c17-9964-4c55-8f6c-30c6511f7048-kube-api-access-whp7n\") pod \"26324c17-9964-4c55-8f6c-30c6511f7048\" (UID: \"26324c17-9964-4c55-8f6c-30c6511f7048\") "
Nov 03 11:15:18.874734 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.874610    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26324c17-9964-4c55-8f6c-30c6511f7048-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26324c17-9964-4c55-8f6c-30c6511f7048-maas-api-service-ca\") pod \"26324c17-9964-4c55-8f6c-30c6511f7048\" (UID: \"26324c17-9964-4c55-8f6c-30c6511f7048\") "
Nov 03 11:15:18.875015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.874993    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26324c17-9964-4c55-8f6c-30c6511f7048-maas-api-service-ca" pod "26324c17-9964-4c55-8f6c-30c6511f7048" (UID: "26324c17-9964-4c55-8f6c-30c6511f7048"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:18.875375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.875356    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26324c17-9964-4c55-8f6c-30c6511f7048-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:18.877046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.877027    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26324c17-9964-4c55-8f6c-30c6511f7048-kube-api-access-whp7n" pod "26324c17-9964-4c55-8f6c-30c6511f7048" (UID: "26324c17-9964-4c55-8f6c-30c6511f7048"). InnerVolumeSpecName "kube-api-access-whp7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:18.977859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:18.977822    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whp7n\" (UniqueName: \"kubernetes.io/projected/26324c17-9964-4c55-8f6c-30c6511f7048-kube-api-access-whp7n\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:19.185994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.184204    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:19.189334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.189068    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:19.225257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.225228    5393 scope.go:122] "RemoveContainer" containerID="c91a65535eca8d272b7df491d20c6e9fec335525fb4e1fcec632673a7658b7ce"
Nov 03 11:15:19.226091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.226070    5393 scope.go:122] "RemoveContainer" containerID="c25ac1fbab87c2fa4acfee3a57e8f9765152d3a2d0687aaf115ada20c058e2fc"
Nov 03 11:15:19.282288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.282246    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f191c99e-0c1a-4f16-9a57-231634a6cd6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191c99e-0c1a-4f16-9a57-231634a6cd6a-maas-api-service-ca\") pod \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\" (UID: \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\") "
Nov 03 11:15:19.282486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.282340    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-kube-api-access-sppwj\" (UniqueName: \"kubernetes.io/projected/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-kube-api-access-sppwj\") pod \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\" (UID: \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\") "
Nov 03 11:15:19.282486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.282409    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f191c99e-0c1a-4f16-9a57-231634a6cd6a-kube-api-access-z76t9\" (UniqueName: \"kubernetes.io/projected/f191c99e-0c1a-4f16-9a57-231634a6cd6a-kube-api-access-z76t9\") pod \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\" (UID: \"f191c99e-0c1a-4f16-9a57-231634a6cd6a\") "
Nov 03 11:15:19.282486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.282439    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-maas-api-service-ca\") pod \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\" (UID: \"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5\") "
Nov 03 11:15:19.283563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.283539    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f191c99e-0c1a-4f16-9a57-231634a6cd6a-maas-api-service-ca" pod "f191c99e-0c1a-4f16-9a57-231634a6cd6a" (UID: "f191c99e-0c1a-4f16-9a57-231634a6cd6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:19.285071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.285039    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-maas-api-service-ca" pod "1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" (UID: "1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:19.295732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.295703    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-kube-api-access-sppwj" pod "1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" (UID: "1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5"). InnerVolumeSpecName "kube-api-access-sppwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:19.295845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.295764    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f191c99e-0c1a-4f16-9a57-231634a6cd6a-kube-api-access-z76t9" pod "f191c99e-0c1a-4f16-9a57-231634a6cd6a" (UID: "f191c99e-0c1a-4f16-9a57-231634a6cd6a"). InnerVolumeSpecName "kube-api-access-z76t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:19.296212 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.296083    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" event={"ID":"74bea47f-c0e1-4a14-acd0-920b5b43e4b3","Type":"ContainerStarted","Data":"ac6d3c2366d196c38840524f50983df063e39b2d4bd891ebb0ea1dc330c29608"}
Nov 03 11:15:19.299787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.299758    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" event={"ID":"51df457b-9f93-446b-8f3a-2d4dd62cfe92","Type":"ContainerStarted","Data":"5c0ec32e97d7ecf78f3fa833bccf0b8e5c4ca7476b3b7e5882c95baaa960ef5c"}
Nov 03 11:15:19.301720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.301697    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6" event={"ID":"b8cac8cf-3549-470b-b998-5cf85a1343bc","Type":"ContainerStarted","Data":"c15f54ccc10ae20598756cce00b6b337996961e985d7f6f9ea3f0a3a5081430b"}
Nov 03 11:15:19.304175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.303614    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"
Nov 03 11:15:19.304175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.303641    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9" event={"ID":"26324c17-9964-4c55-8f6c-30c6511f7048","Type":"ContainerDied","Data":"b65d397f03f6b9ca48dfcf4131d92805a480217cf297707a6f6357f78f7bca98"}
Nov 03 11:15:19.304175 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.303694    5393 scope.go:122] "RemoveContainer" containerID="d89f53249ac243c78443963b697a1818dc70a2ed6db9ea371029813f8de9bbac"
Nov 03 11:15:19.307666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.307637    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd" event={"ID":"f191c99e-0c1a-4f16-9a57-231634a6cd6a","Type":"ContainerDied","Data":"5b0d78aa25b2988a00b73e890afd3aea1d71bc2f3bbd066a230822a673e9f5a5"}
Nov 03 11:15:19.307795 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.307731    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"
Nov 03 11:15:19.312645 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.312623    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr" event={"ID":"1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5","Type":"ContainerDied","Data":"859fc22756d3094817ec39a97c30731dbc66ec6748bc9124408ba944bddf2cb5"}
Nov 03 11:15:19.312790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.312643    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"
Nov 03 11:15:19.315719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.315691    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" event={"ID":"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271","Type":"ContainerStarted","Data":"7192e9a91cc4d71af31151e3db70884bcbb5f6f9b3b73ce5cdf9697021c26269"}
Nov 03 11:15:19.319256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.319173    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4" event={"ID":"e3d86bb9-ee25-410c-99d3-adc646178897","Type":"ContainerStarted","Data":"5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496"}
Nov 03 11:15:19.319256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.319225    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4" event={"ID":"e3d86bb9-ee25-410c-99d3-adc646178897","Type":"ContainerStarted","Data":"066e8560ff324e239dfeb42c0572d0939481eeafe19d82db75695ca08ec64240"}
Nov 03 11:15:19.319423 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.319282    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:19.322522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.322497    5393 generic.go:356] "Generic (PLEG): container finished" podID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerID="b092211f9d7c52071a71da72546d505506e3547fde9bc7aeffd7b23872305142" exitCode=60
Nov 03 11:15:19.322635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.322534    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" event={"ID":"2e754a45-81bc-45e1-9949-5c8c9f6a638c","Type":"ContainerDied","Data":"b092211f9d7c52071a71da72546d505506e3547fde9bc7aeffd7b23872305142"}
Nov 03 11:15:19.322973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.322948    5393 scope.go:122] "RemoveContainer" containerID="b092211f9d7c52071a71da72546d505506e3547fde9bc7aeffd7b23872305142"
Nov 03 11:15:19.323307 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:19.323284    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk_odh-ai-gateway-infra(2e754a45-81bc-45e1-9949-5c8c9f6a638c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c"
Nov 03 11:15:19.327540 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.327521    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"]
Nov 03 11:15:19.378643 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.369600    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4" podStartSLOduration=2.369582922 podStartE2EDuration="2.369582922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:19.365076319 +0000 UTC m=+2763.269930762" watchObservedRunningTime="2026-11-03 11:15:19.369582922 +0000 UTC m=+2763.274437364"
Nov 03 11:15:19.399372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.399336    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191c99e-0c1a-4f16-9a57-231634a6cd6a-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:19.399579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.399564    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppwj\" (UniqueName: \"kubernetes.io/projected/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-kube-api-access-sppwj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:19.399681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.399658    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z76t9\" (UniqueName: \"kubernetes.io/projected/f191c99e-0c1a-4f16-9a57-231634a6cd6a-kube-api-access-z76t9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:19.399845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.399685    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:19.441099 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.441076    5393 scope.go:122] "RemoveContainer" containerID="c9514698b25c42cedf5f09e26bfb7ee0ffddfb94891933dd1e733ae9b5c22f56"
Nov 03 11:15:19.460304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.460279    5393 scope.go:122] "RemoveContainer" containerID="c82aa6c6f3c211ca8977a97e43148cb5904dc2f3945b99251f70b2e48d748e28"
Nov 03 11:15:19.471119 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.471086    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"]
Nov 03 11:15:19.471812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.471793    5393 scope.go:122] "RemoveContainer" containerID="bd8618b8c6008f4d11054edd3cd81e0ef7d6c904ff28926dc09df695431abdfd"
Nov 03 11:15:19.476467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.476446    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-fcc2ce-29895075-8pbmd"]
Nov 03 11:15:19.530328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.530287    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"]
Nov 03 11:15:19.552398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.552359    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5711cf-29895075-cnbv9"]
Nov 03 11:15:19.563123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.561752    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"]
Nov 03 11:15:19.569092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:19.567630    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-185908-29895075-fg8nr"]
Nov 03 11:15:20.234110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.234052    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" path="/var/lib/kubelet/pods/1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5/volumes"
Nov 03 11:15:20.234564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.234547    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" path="/var/lib/kubelet/pods/26324c17-9964-4c55-8f6c-30c6511f7048/volumes"
Nov 03 11:15:20.235166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.235149    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" path="/var/lib/kubelet/pods/f191c99e-0c1a-4f16-9a57-231634a6cd6a/volumes"
Nov 03 11:15:20.319125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.319086    5393 scope.go:122] "RemoveContainer" containerID="e851a9e473c1a0235b7b0b541816e5aa63871efe22f251b8a34c02ef255b9c49"
Nov 03 11:15:20.328488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.328463    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"]
Nov 03 11:15:20.352745 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.352708    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" event={"ID":"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81","Type":"ContainerStarted","Data":"c8f34484b138f8840f67611f615b11c415c058a231eae12914a80d75df082309"}
Nov 03 11:15:20.353033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.353011    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"]
Nov 03 11:15:20.360276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.359274    5393 generic.go:356] "Generic (PLEG): container finished" podID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerID="a168a8bbd0e586c474f2074108cf9fdf8c0d675fe55561ebb9d1fe17306d252c" exitCode=60
Nov 03 11:15:20.360276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.359372    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" event={"ID":"763c6d00-52d4-4cc5-af13-1e7cfa1fc700","Type":"ContainerDied","Data":"a168a8bbd0e586c474f2074108cf9fdf8c0d675fe55561ebb9d1fe17306d252c"}
Nov 03 11:15:20.360276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.359429    5393 scope.go:122] "RemoveContainer" containerID="db9680eda33a82d2bd7cafe381acb6dae800f217a6929879e8a0df883b3761f2"
Nov 03 11:15:20.360276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.359903    5393 scope.go:122] "RemoveContainer" containerID="a168a8bbd0e586c474f2074108cf9fdf8c0d675fe55561ebb9d1fe17306d252c"
Nov 03 11:15:20.360826 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:20.360396    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6_odh-ai-gateway-infra(763c6d00-52d4-4cc5-af13-1e7cfa1fc700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700"
Nov 03 11:15:20.362110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.362013    5393 generic.go:356] "Generic (PLEG): container finished" podID="1107f46f-c86c-4569-b755-9aa28e256af0" containerID="1da958e721d7df62d41066c7a4b80456758f835525395e5b7db4c13997dbb33a" exitCode=60
Nov 03 11:15:20.362234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.362132    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" event={"ID":"1107f46f-c86c-4569-b755-9aa28e256af0","Type":"ContainerDied","Data":"1da958e721d7df62d41066c7a4b80456758f835525395e5b7db4c13997dbb33a"}
Nov 03 11:15:20.362593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.362577    5393 scope.go:122] "RemoveContainer" containerID="1da958e721d7df62d41066c7a4b80456758f835525395e5b7db4c13997dbb33a"
Nov 03 11:15:20.362997 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:20.362975    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n_odh-ai-gateway-infra(1107f46f-c86c-4569-b755-9aa28e256af0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" podUID="1107f46f-c86c-4569-b755-9aa28e256af0"
Nov 03 11:15:20.364896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.364873    5393 generic.go:356] "Generic (PLEG): container finished" podID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerID="fb7962dcfce4af3b3d3dca8ea310cc5c66dba1855bd6e1ce5dd0cd68fddfbccc" exitCode=60
Nov 03 11:15:20.365013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.364979    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" event={"ID":"36bf3819-7aea-4e1c-ad10-66f59268dcf1","Type":"ContainerDied","Data":"fb7962dcfce4af3b3d3dca8ea310cc5c66dba1855bd6e1ce5dd0cd68fddfbccc"}
Nov 03 11:15:20.366049 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.365598    5393 scope.go:122] "RemoveContainer" containerID="fb7962dcfce4af3b3d3dca8ea310cc5c66dba1855bd6e1ce5dd0cd68fddfbccc"
Nov 03 11:15:20.366049 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:20.365985    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl_odh-ai-gateway-infra(36bf3819-7aea-4e1c-ad10-66f59268dcf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1"
Nov 03 11:15:20.367907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.367881    5393 generic.go:356] "Generic (PLEG): container finished" podID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerID="0746b6b234694cb65c17f5d09e14d39f9c42f8a2ad68b97bb70e95dd4d332ab1" exitCode=60
Nov 03 11:15:20.368030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.367984    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" event={"ID":"b5cea438-8f15-4414-8a1c-7f6df195d5a7","Type":"ContainerDied","Data":"0746b6b234694cb65c17f5d09e14d39f9c42f8a2ad68b97bb70e95dd4d332ab1"}
Nov 03 11:15:20.368447 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.368424    5393 scope.go:122] "RemoveContainer" containerID="0746b6b234694cb65c17f5d09e14d39f9c42f8a2ad68b97bb70e95dd4d332ab1"
Nov 03 11:15:20.369553 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:20.368828    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj_odh-ai-gateway-infra(b5cea438-8f15-4414-8a1c-7f6df195d5a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7"
Nov 03 11:15:20.370289 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.370206    5393 generic.go:356] "Generic (PLEG): container finished" podID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerID="7192e9a91cc4d71af31151e3db70884bcbb5f6f9b3b73ce5cdf9697021c26269" exitCode=60
Nov 03 11:15:20.370344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.370306    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" event={"ID":"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271","Type":"ContainerDied","Data":"7192e9a91cc4d71af31151e3db70884bcbb5f6f9b3b73ce5cdf9697021c26269"}
Nov 03 11:15:20.370760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.370737    5393 scope.go:122] "RemoveContainer" containerID="7192e9a91cc4d71af31151e3db70884bcbb5f6f9b3b73ce5cdf9697021c26269"
Nov 03 11:15:20.371343 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:20.371195    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf_odh-ai-gateway-infra(fb58eeb4-c1d2-449f-90d7-f92ab4eb1271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271"
Nov 03 11:15:20.376416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.376389    5393 generic.go:356] "Generic (PLEG): container finished" podID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerID="ac6d3c2366d196c38840524f50983df063e39b2d4bd891ebb0ea1dc330c29608" exitCode=60
Nov 03 11:15:20.377763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.376540    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" event={"ID":"74bea47f-c0e1-4a14-acd0-920b5b43e4b3","Type":"ContainerDied","Data":"ac6d3c2366d196c38840524f50983df063e39b2d4bd891ebb0ea1dc330c29608"}
Nov 03 11:15:20.383716 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.383648    5393 generic.go:356] "Generic (PLEG): container finished" podID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerID="5c0ec32e97d7ecf78f3fa833bccf0b8e5c4ca7476b3b7e5882c95baaa960ef5c" exitCode=60
Nov 03 11:15:20.383820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.383777    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" event={"ID":"51df457b-9f93-446b-8f3a-2d4dd62cfe92","Type":"ContainerDied","Data":"5c0ec32e97d7ecf78f3fa833bccf0b8e5c4ca7476b3b7e5882c95baaa960ef5c"}
Nov 03 11:15:20.386609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.386590    5393 generic.go:356] "Generic (PLEG): container finished" podID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerID="735c5b113f31b34e4fd1762761a5e2d2fd79b920a1ce211bb4e8801050505214" exitCode=60
Nov 03 11:15:20.386713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.386663    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" event={"ID":"2534e7e5-ea82-432e-af50-cc00bb0fa611","Type":"ContainerDied","Data":"735c5b113f31b34e4fd1762761a5e2d2fd79b920a1ce211bb4e8801050505214"}
Nov 03 11:15:20.387266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.387247    5393 scope.go:122] "RemoveContainer" containerID="735c5b113f31b34e4fd1762761a5e2d2fd79b920a1ce211bb4e8801050505214"
Nov 03 11:15:20.388852 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:20.388825    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg_odh-ai-gateway-infra(2534e7e5-ea82-432e-af50-cc00bb0fa611)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611"
Nov 03 11:15:20.389261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.389237    5393 scope.go:122] "RemoveContainer" containerID="52b302ca7302c772ece0a02343f8046908ad2cdf2c3801566f56f937bc600bf9"
Nov 03 11:15:20.390710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.390668    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6" event={"ID":"b8cac8cf-3549-470b-b998-5cf85a1343bc","Type":"ContainerStarted","Data":"d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936"}
Nov 03 11:15:20.391135 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.391115    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:20.394337 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.393876    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" event={"ID":"40de722e-3cd0-4923-9aec-bc5695a58f9e","Type":"ContainerStarted","Data":"90db08a20d26c457e31cbf395c76ef5a931d2035225e23733a9533da552345d5"}
Nov 03 11:15:20.419668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.419599    5393 scope.go:122] "RemoveContainer" containerID="7852ca8df5093500e739e754d6ec8a2567acc5482ffe808f8dcb9f923941c2e1"
Nov 03 11:15:20.458528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.458487    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"]
Nov 03 11:15:20.572236 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.572199    5393 scope.go:122] "RemoveContainer" containerID="04250294c4649d122fb0b3bdee781ccfab7e0fe31f9a37b24c2bb0daefc64a01"
Nov 03 11:15:20.663492 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.663465    5393 scope.go:122] "RemoveContainer" containerID="9ffdb64da60a48cd7c5d43939467dc3c3cbdcbff174993852ca54928ae7c0a3a"
Nov 03 11:15:20.676532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.676468    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6" podStartSLOduration=3.6764492300000002 podStartE2EDuration="3.67644923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:20.66888085 +0000 UTC m=+2764.573735293" watchObservedRunningTime="2026-11-03 11:15:20.67644923 +0000 UTC m=+2764.581303671"
Nov 03 11:15:20.686208 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.686183    5393 scope.go:122] "RemoveContainer" containerID="3ed3ac1d878b83c9d0d5dd06b25c55efa7e52a6997aeca708ec4b2be37477b37"
Nov 03 11:15:20.736475 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.736071    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:20.747823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.747710    5393 scope.go:122] "RemoveContainer" containerID="c6236764a34e8e8be01cb06b7b9b8a0cae5ba527754fda885439190a66524816"
Nov 03 11:15:20.748807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.748771    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:20.773817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.773795    5393 scope.go:122] "RemoveContainer" containerID="b2dc7e3dd1668dd0d58ccd46cf9e62786f237c95bf0484d9b9276e6d5a3500a6"
Nov 03 11:15:20.789524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.787717    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-maas-api-service-ca\") pod \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\" (UID: \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\") "
Nov 03 11:15:20.789524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.787939    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-kube-api-access-qt662\" (UniqueName: \"kubernetes.io/projected/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-kube-api-access-qt662\") pod \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\" (UID: \"74bea47f-c0e1-4a14-acd0-920b5b43e4b3\") "
Nov 03 11:15:20.789524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.788086    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e754a45-81bc-45e1-9949-5c8c9f6a638c-kube-api-access-542lf\" (UniqueName: \"kubernetes.io/projected/2e754a45-81bc-45e1-9949-5c8c9f6a638c-kube-api-access-542lf\") pod \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\" (UID: \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\") "
Nov 03 11:15:20.789524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.788204    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e754a45-81bc-45e1-9949-5c8c9f6a638c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e754a45-81bc-45e1-9949-5c8c9f6a638c-maas-api-service-ca\") pod \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\" (UID: \"2e754a45-81bc-45e1-9949-5c8c9f6a638c\") "
Nov 03 11:15:20.790176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.790146    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e754a45-81bc-45e1-9949-5c8c9f6a638c-maas-api-service-ca" pod "2e754a45-81bc-45e1-9949-5c8c9f6a638c" (UID: "2e754a45-81bc-45e1-9949-5c8c9f6a638c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:20.790660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.790635    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-maas-api-service-ca" pod "74bea47f-c0e1-4a14-acd0-920b5b43e4b3" (UID: "74bea47f-c0e1-4a14-acd0-920b5b43e4b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:20.884668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.884298    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:20.886765 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.886719    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-kube-api-access-qt662" pod "74bea47f-c0e1-4a14-acd0-920b5b43e4b3" (UID: "74bea47f-c0e1-4a14-acd0-920b5b43e4b3"). InnerVolumeSpecName "kube-api-access-qt662". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:20.890993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.890909    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.890993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.890954    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt662\" (UniqueName: \"kubernetes.io/projected/74bea47f-c0e1-4a14-acd0-920b5b43e4b3-kube-api-access-qt662\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.890993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.890972    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e754a45-81bc-45e1-9949-5c8c9f6a638c-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.893798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.893131    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e754a45-81bc-45e1-9949-5c8c9f6a638c-kube-api-access-542lf" pod "2e754a45-81bc-45e1-9949-5c8c9f6a638c" (UID: "2e754a45-81bc-45e1-9949-5c8c9f6a638c"). InnerVolumeSpecName "kube-api-access-542lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:20.992346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.992258    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51df457b-9f93-446b-8f3a-2d4dd62cfe92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51df457b-9f93-446b-8f3a-2d4dd62cfe92-maas-api-service-ca\") pod \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\" (UID: \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\") "
Nov 03 11:15:20.992530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.992510    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51df457b-9f93-446b-8f3a-2d4dd62cfe92-kube-api-access-rdscx\" (UniqueName: \"kubernetes.io/projected/51df457b-9f93-446b-8f3a-2d4dd62cfe92-kube-api-access-rdscx\") pod \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\" (UID: \"51df457b-9f93-446b-8f3a-2d4dd62cfe92\") "
Nov 03 11:15:20.992651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.992626    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51df457b-9f93-446b-8f3a-2d4dd62cfe92-maas-api-service-ca" pod "51df457b-9f93-446b-8f3a-2d4dd62cfe92" (UID: "51df457b-9f93-446b-8f3a-2d4dd62cfe92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:20.993344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.993322    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-542lf\" (UniqueName: \"kubernetes.io/projected/2e754a45-81bc-45e1-9949-5c8c9f6a638c-kube-api-access-542lf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.993423 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.993346    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51df457b-9f93-446b-8f3a-2d4dd62cfe92-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:20.994951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:20.994906    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51df457b-9f93-446b-8f3a-2d4dd62cfe92-kube-api-access-rdscx" pod "51df457b-9f93-446b-8f3a-2d4dd62cfe92" (UID: "51df457b-9f93-446b-8f3a-2d4dd62cfe92"). InnerVolumeSpecName "kube-api-access-rdscx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:21.095162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.095119    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdscx\" (UniqueName: \"kubernetes.io/projected/51df457b-9f93-446b-8f3a-2d4dd62cfe92-kube-api-access-rdscx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.385327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.385293    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"]
Nov 03 11:15:21.460523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.460383    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"]
Nov 03 11:15:21.468091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.468041    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"
Nov 03 11:15:21.468091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.468067    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk" event={"ID":"2e754a45-81bc-45e1-9949-5c8c9f6a638c","Type":"ContainerDied","Data":"ebdcaac6390e4449e26a087cfddbe491b1a25a824f434e3866602bb986d1358b"}
Nov 03 11:15:21.468238 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.468122    5393 scope.go:122] "RemoveContainer" containerID="b092211f9d7c52071a71da72546d505506e3547fde9bc7aeffd7b23872305142"
Nov 03 11:15:21.472499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.472412    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5" event={"ID":"74bea47f-c0e1-4a14-acd0-920b5b43e4b3","Type":"ContainerDied","Data":"02011c39cba6030fcda7165a3df5ac1d9a94e36006d691ec5fec70596ed774c3"}
Nov 03 11:15:21.473212 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.472995    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"
Nov 03 11:15:21.474096 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.473539    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"]
Nov 03 11:15:21.477252 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.474958    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq" event={"ID":"51df457b-9f93-446b-8f3a-2d4dd62cfe92","Type":"ContainerDied","Data":"abca26aaf7d72c9775af37a3ba56f8b5386f4db9ec78da6c650848aaa2b91a58"}
Nov 03 11:15:21.477252 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.475047    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"
Nov 03 11:15:21.483618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.483594    5393 generic.go:356] "Generic (PLEG): container finished" podID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerID="90db08a20d26c457e31cbf395c76ef5a931d2035225e23733a9533da552345d5" exitCode=60
Nov 03 11:15:21.483733 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.483717    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" event={"ID":"40de722e-3cd0-4923-9aec-bc5695a58f9e","Type":"ContainerDied","Data":"90db08a20d26c457e31cbf395c76ef5a931d2035225e23733a9533da552345d5"}
Nov 03 11:15:21.484156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.484133    5393 scope.go:122] "RemoveContainer" containerID="90db08a20d26c457e31cbf395c76ef5a931d2035225e23733a9533da552345d5"
Nov 03 11:15:21.484610 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:21.484519    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b_odh-ai-gateway-infra(40de722e-3cd0-4923-9aec-bc5695a58f9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e"
Nov 03 11:15:21.489707 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.489452    5393 generic.go:356] "Generic (PLEG): container finished" podID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerID="c8f34484b138f8840f67611f615b11c415c058a231eae12914a80d75df082309" exitCode=60
Nov 03 11:15:21.489707 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.489657    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" event={"ID":"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81","Type":"ContainerDied","Data":"c8f34484b138f8840f67611f615b11c415c058a231eae12914a80d75df082309"}
Nov 03 11:15:21.508824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.508751    5393 scope.go:122] "RemoveContainer" containerID="ac6d3c2366d196c38840524f50983df063e39b2d4bd891ebb0ea1dc330c29608"
Nov 03 11:15:21.542167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.542129    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"]
Nov 03 11:15:21.549561 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.549536    5393 scope.go:122] "RemoveContainer" containerID="5c0ec32e97d7ecf78f3fa833bccf0b8e5c4ca7476b3b7e5882c95baaa960ef5c"
Nov 03 11:15:21.592099 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.592066    5393 scope.go:122] "RemoveContainer" containerID="c25ac1fbab87c2fa4acfee3a57e8f9765152d3a2d0687aaf115ada20c058e2fc"
Nov 03 11:15:21.597061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.597035    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"]
Nov 03 11:15:21.617305 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.616012    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce5555-29895075-x6ntk"]
Nov 03 11:15:21.642556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.642168    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"]
Nov 03 11:15:21.666758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.666431    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"]
Nov 03 11:15:21.674974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.674887    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"]
Nov 03 11:15:21.687080 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.687046    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895075-k2rd5"]
Nov 03 11:15:21.688039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.688010    5393 scope.go:122] "RemoveContainer" containerID="c91a65535eca8d272b7df491d20c6e9fec335525fb4e1fcec632673a7658b7ce"
Nov 03 11:15:21.694695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.694668    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"]
Nov 03 11:15:21.743534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.743484    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"]
Nov 03 11:15:21.753657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.753595    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-ef1662-29895075-thhvq"]
Nov 03 11:15:21.840968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.839893    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:21.858273 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.858249    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:21.909653 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.909568    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cea438-8f15-4414-8a1c-7f6df195d5a7-kube-api-access-ctqn5\" (UniqueName: \"kubernetes.io/projected/b5cea438-8f15-4414-8a1c-7f6df195d5a7-kube-api-access-ctqn5\") pod \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\" (UID: \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\") "
Nov 03 11:15:21.909963 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.909941    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-kube-api-access-m8qj6\") pod \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\" (UID: \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\") "
Nov 03 11:15:21.910263 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.910248    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-maas-api-service-ca\") pod \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\" (UID: \"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271\") "
Nov 03 11:15:21.910524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.910511    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5cea438-8f15-4414-8a1c-7f6df195d5a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cea438-8f15-4414-8a1c-7f6df195d5a7-maas-api-service-ca\") pod \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\" (UID: \"b5cea438-8f15-4414-8a1c-7f6df195d5a7\") "
Nov 03 11:15:21.922994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.922123    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-maas-api-service-ca" pod "fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" (UID: "fb58eeb4-c1d2-449f-90d7-f92ab4eb1271"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:21.922994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.922637    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-kube-api-access-m8qj6" pod "fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" (UID: "fb58eeb4-c1d2-449f-90d7-f92ab4eb1271"). InnerVolumeSpecName "kube-api-access-m8qj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:21.922994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.922820    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cea438-8f15-4414-8a1c-7f6df195d5a7-maas-api-service-ca" pod "b5cea438-8f15-4414-8a1c-7f6df195d5a7" (UID: "b5cea438-8f15-4414-8a1c-7f6df195d5a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:21.924052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.923852    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cea438-8f15-4414-8a1c-7f6df195d5a7-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.924052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.923882    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-kube-api-access-m8qj6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.924052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.924028    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:21.926992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:21.926968    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cea438-8f15-4414-8a1c-7f6df195d5a7-kube-api-access-ctqn5" pod "b5cea438-8f15-4414-8a1c-7f6df195d5a7" (UID: "b5cea438-8f15-4414-8a1c-7f6df195d5a7"). InnerVolumeSpecName "kube-api-access-ctqn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.027647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.027574    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctqn5\" (UniqueName: \"kubernetes.io/projected/b5cea438-8f15-4414-8a1c-7f6df195d5a7-kube-api-access-ctqn5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.055052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.055024    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:22.062505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.062414    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:22.066411 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.066380    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:22.128617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.128518    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36bf3819-7aea-4e1c-ad10-66f59268dcf1-kube-api-access-s5dgr\" (UniqueName: \"kubernetes.io/projected/36bf3819-7aea-4e1c-ad10-66f59268dcf1-kube-api-access-s5dgr\") pod \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\" (UID: \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\") "
Nov 03 11:15:22.128768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.128635    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1107f46f-c86c-4569-b755-9aa28e256af0-kube-api-access-99xrx\" (UniqueName: \"kubernetes.io/projected/1107f46f-c86c-4569-b755-9aa28e256af0-kube-api-access-99xrx\") pod \"1107f46f-c86c-4569-b755-9aa28e256af0\" (UID: \"1107f46f-c86c-4569-b755-9aa28e256af0\") "
Nov 03 11:15:22.128768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.128683    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-kube-api-access-v6w7f\" (UniqueName: \"kubernetes.io/projected/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-kube-api-access-v6w7f\") pod \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\" (UID: \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\") "
Nov 03 11:15:22.128964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.128827    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36bf3819-7aea-4e1c-ad10-66f59268dcf1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf3819-7aea-4e1c-ad10-66f59268dcf1-maas-api-service-ca\") pod \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\" (UID: \"36bf3819-7aea-4e1c-ad10-66f59268dcf1\") "
Nov 03 11:15:22.129245 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.129161    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1107f46f-c86c-4569-b755-9aa28e256af0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1107f46f-c86c-4569-b755-9aa28e256af0-maas-api-service-ca\") pod \"1107f46f-c86c-4569-b755-9aa28e256af0\" (UID: \"1107f46f-c86c-4569-b755-9aa28e256af0\") "
Nov 03 11:15:22.132669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.132456    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-maas-api-service-ca\") pod \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\" (UID: \"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81\") "
Nov 03 11:15:22.134958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.133996    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-maas-api-service-ca" pod "4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" (UID: "4ccb486b-de82-43bc-b8fa-47e5a6a2bf81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.134958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.134513    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1107f46f-c86c-4569-b755-9aa28e256af0-maas-api-service-ca" pod "1107f46f-c86c-4569-b755-9aa28e256af0" (UID: "1107f46f-c86c-4569-b755-9aa28e256af0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.134958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.134759    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36bf3819-7aea-4e1c-ad10-66f59268dcf1-maas-api-service-ca" pod "36bf3819-7aea-4e1c-ad10-66f59268dcf1" (UID: "36bf3819-7aea-4e1c-ad10-66f59268dcf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.137186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.137160    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bf3819-7aea-4e1c-ad10-66f59268dcf1-kube-api-access-s5dgr" pod "36bf3819-7aea-4e1c-ad10-66f59268dcf1" (UID: "36bf3819-7aea-4e1c-ad10-66f59268dcf1"). InnerVolumeSpecName "kube-api-access-s5dgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.137532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.137507    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-kube-api-access-v6w7f" pod "4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" (UID: "4ccb486b-de82-43bc-b8fa-47e5a6a2bf81"). InnerVolumeSpecName "kube-api-access-v6w7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.138057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.138035    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1107f46f-c86c-4569-b755-9aa28e256af0-kube-api-access-99xrx" pod "1107f46f-c86c-4569-b755-9aa28e256af0" (UID: "1107f46f-c86c-4569-b755-9aa28e256af0"). InnerVolumeSpecName "kube-api-access-99xrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.142845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.142824    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:22.148709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.148685    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:22.234984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.234888    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-maas-api-service-ca\") pod \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\" (UID: \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\") "
Nov 03 11:15:22.235146 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.235012    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2534e7e5-ea82-432e-af50-cc00bb0fa611-kube-api-access-k88z7\" (UniqueName: \"kubernetes.io/projected/2534e7e5-ea82-432e-af50-cc00bb0fa611-kube-api-access-k88z7\") pod \"2534e7e5-ea82-432e-af50-cc00bb0fa611\" (UID: \"2534e7e5-ea82-432e-af50-cc00bb0fa611\") "
Nov 03 11:15:22.235146 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.235113    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2534e7e5-ea82-432e-af50-cc00bb0fa611-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2534e7e5-ea82-432e-af50-cc00bb0fa611-maas-api-service-ca\") pod \"2534e7e5-ea82-432e-af50-cc00bb0fa611\" (UID: \"2534e7e5-ea82-432e-af50-cc00bb0fa611\") "
Nov 03 11:15:22.235372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.235341    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-maas-api-service-ca" pod "763c6d00-52d4-4cc5-af13-1e7cfa1fc700" (UID: "763c6d00-52d4-4cc5-af13-1e7cfa1fc700"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.235462 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.235437    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-kube-api-access-9bdhq\" (UniqueName: \"kubernetes.io/projected/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-kube-api-access-9bdhq\") pod \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\" (UID: \"763c6d00-52d4-4cc5-af13-1e7cfa1fc700\") "
Nov 03 11:15:22.235857 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.235824    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2534e7e5-ea82-432e-af50-cc00bb0fa611-maas-api-service-ca" pod "2534e7e5-ea82-432e-af50-cc00bb0fa611" (UID: "2534e7e5-ea82-432e-af50-cc00bb0fa611"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.236375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.236351    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.236474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.236381    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1107f46f-c86c-4569-b755-9aa28e256af0-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.236474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.236397    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.236474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.236412    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5dgr\" (UniqueName: \"kubernetes.io/projected/36bf3819-7aea-4e1c-ad10-66f59268dcf1-kube-api-access-s5dgr\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.236474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.236426    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2534e7e5-ea82-432e-af50-cc00bb0fa611-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.236474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.236440    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xrx\" (UniqueName: \"kubernetes.io/projected/1107f46f-c86c-4569-b755-9aa28e256af0-kube-api-access-99xrx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.236474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.236454    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6w7f\" (UniqueName: \"kubernetes.io/projected/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81-kube-api-access-v6w7f\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.236474 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.236469    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf3819-7aea-4e1c-ad10-66f59268dcf1-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.238161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.238134    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-kube-api-access-9bdhq" pod "763c6d00-52d4-4cc5-af13-1e7cfa1fc700" (UID: "763c6d00-52d4-4cc5-af13-1e7cfa1fc700"). InnerVolumeSpecName "kube-api-access-9bdhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.238294 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.238272    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" path="/var/lib/kubelet/pods/2e754a45-81bc-45e1-9949-5c8c9f6a638c/volumes"
Nov 03 11:15:22.238803 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.238782    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2534e7e5-ea82-432e-af50-cc00bb0fa611-kube-api-access-k88z7" pod "2534e7e5-ea82-432e-af50-cc00bb0fa611" (UID: "2534e7e5-ea82-432e-af50-cc00bb0fa611"). InnerVolumeSpecName "kube-api-access-k88z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.239010 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.238990    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" path="/var/lib/kubelet/pods/51df457b-9f93-446b-8f3a-2d4dd62cfe92/volumes"
Nov 03 11:15:22.239616 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.239596    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" path="/var/lib/kubelet/pods/74bea47f-c0e1-4a14-acd0-920b5b43e4b3/volumes"
Nov 03 11:15:22.338168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.338137    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88z7\" (UniqueName: \"kubernetes.io/projected/2534e7e5-ea82-432e-af50-cc00bb0fa611-kube-api-access-k88z7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.338168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.338165    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bdhq\" (UniqueName: \"kubernetes.io/projected/763c6d00-52d4-4cc5-af13-1e7cfa1fc700-kube-api-access-9bdhq\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.516778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.516750    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"
Nov 03 11:15:22.517490 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.517468    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq" event={"ID":"4ccb486b-de82-43bc-b8fa-47e5a6a2bf81","Type":"ContainerDied","Data":"fe3393a39222d19fc68aa44d3b5a027b5d3230f7c67469ac3caea259536607fb"}
Nov 03 11:15:22.517572 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.517524    5393 scope.go:122] "RemoveContainer" containerID="c8f34484b138f8840f67611f615b11c415c058a231eae12914a80d75df082309"
Nov 03 11:15:22.519262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.519187    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"
Nov 03 11:15:22.519262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.519208    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6" event={"ID":"763c6d00-52d4-4cc5-af13-1e7cfa1fc700","Type":"ContainerDied","Data":"3345c6c1a6ffe4c9086cea59601e8b0dfd4783547cfda4996fc294e7f2c412f9"}
Nov 03 11:15:22.521008 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.520987    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"
Nov 03 11:15:22.521120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.521012    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n" event={"ID":"1107f46f-c86c-4569-b755-9aa28e256af0","Type":"ContainerDied","Data":"c0d5947e5bb29764f4c41d999302e39b8054deae577c11cee8a8ac48de5b754c"}
Nov 03 11:15:22.522811 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.522789    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"
Nov 03 11:15:22.522811 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.522801    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl" event={"ID":"36bf3819-7aea-4e1c-ad10-66f59268dcf1","Type":"ContainerDied","Data":"7f6f30b6d80a865390c2b89312c6b84de162697eb6f583f24a637454d727c4d6"}
Nov 03 11:15:22.524319 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.524293    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"
Nov 03 11:15:22.524430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.524323    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj" event={"ID":"b5cea438-8f15-4414-8a1c-7f6df195d5a7","Type":"ContainerDied","Data":"dcbc23210c82fb5899f7232649bdf032cbfa64d25c5938966a89d49d501e3515"}
Nov 03 11:15:22.526055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.526035    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf" event={"ID":"fb58eeb4-c1d2-449f-90d7-f92ab4eb1271","Type":"ContainerDied","Data":"807fc7db27534b9b6a8bfbcda7459ea042aa85d7ff9a28ff913e507bff3856e9"}
Nov 03 11:15:22.526154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.526103    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"
Nov 03 11:15:22.535095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.534340    5393 scope.go:122] "RemoveContainer" containerID="a168a8bbd0e586c474f2074108cf9fdf8c0d675fe55561ebb9d1fe17306d252c"
Nov 03 11:15:22.535095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.534411    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg" event={"ID":"2534e7e5-ea82-432e-af50-cc00bb0fa611","Type":"ContainerDied","Data":"e3a9de48c839d5a23b800f22dff1d65f2f03951ab735a3c307e32ae288dd2ef7"}
Nov 03 11:15:22.535095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.534459    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"
Nov 03 11:15:22.547733 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.547714    5393 scope.go:122] "RemoveContainer" containerID="1da958e721d7df62d41066c7a4b80456758f835525395e5b7db4c13997dbb33a"
Nov 03 11:15:22.555996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.555964    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"]
Nov 03 11:15:22.561108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.561072    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b76c62-29895075-nsvq6"]
Nov 03 11:15:22.561177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.561163    5393 scope.go:122] "RemoveContainer" containerID="fb7962dcfce4af3b3d3dca8ea310cc5c66dba1855bd6e1ce5dd0cd68fddfbccc"
Nov 03 11:15:22.574176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.574154    5393 scope.go:122] "RemoveContainer" containerID="0746b6b234694cb65c17f5d09e14d39f9c42f8a2ad68b97bb70e95dd4d332ab1"
Nov 03 11:15:22.586879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.586860    5393 scope.go:122] "RemoveContainer" containerID="7192e9a91cc4d71af31151e3db70884bcbb5f6f9b3b73ce5cdf9697021c26269"
Nov 03 11:15:22.592213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.592188    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"]
Nov 03 11:15:22.599111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.599082    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895075-lprdq"]
Nov 03 11:15:22.617178 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.617143    5393 scope.go:122] "RemoveContainer" containerID="735c5b113f31b34e4fd1762761a5e2d2fd79b920a1ce211bb4e8801050505214"
Nov 03 11:15:22.651499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.651459    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"]
Nov 03 11:15:22.662105 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.662064    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-47a8a4-29895075-8nphj"]
Nov 03 11:15:22.669109 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.669066    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"]
Nov 03 11:15:22.680541 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.680507    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e92a08-29895075-c2f9n"]
Nov 03 11:15:22.681818 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.681793    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"]
Nov 03 11:15:22.689889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.689865    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895075-bdqjl"]
Nov 03 11:15:22.697219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.697191    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"]
Nov 03 11:15:22.702547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.702512    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:22.703079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.703063    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-9c53b2-29895075-c98lf"]
Nov 03 11:15:22.707649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.707630    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"]
Nov 03 11:15:22.711626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.711599    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a68103-29895075-brwzg"]
Nov 03 11:15:22.752616 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.752574    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40de722e-3cd0-4923-9aec-bc5695a58f9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40de722e-3cd0-4923-9aec-bc5695a58f9e-maas-api-service-ca\") pod \"40de722e-3cd0-4923-9aec-bc5695a58f9e\" (UID: \"40de722e-3cd0-4923-9aec-bc5695a58f9e\") "
Nov 03 11:15:22.752823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.752743    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40de722e-3cd0-4923-9aec-bc5695a58f9e-kube-api-access-t5znx\" (UniqueName: \"kubernetes.io/projected/40de722e-3cd0-4923-9aec-bc5695a58f9e-kube-api-access-t5znx\") pod \"40de722e-3cd0-4923-9aec-bc5695a58f9e\" (UID: \"40de722e-3cd0-4923-9aec-bc5695a58f9e\") "
Nov 03 11:15:22.753003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.752981    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40de722e-3cd0-4923-9aec-bc5695a58f9e-maas-api-service-ca" pod "40de722e-3cd0-4923-9aec-bc5695a58f9e" (UID: "40de722e-3cd0-4923-9aec-bc5695a58f9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:22.753650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.753628    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40de722e-3cd0-4923-9aec-bc5695a58f9e-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:22.755208 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.755184    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40de722e-3cd0-4923-9aec-bc5695a58f9e-kube-api-access-t5znx" pod "40de722e-3cd0-4923-9aec-bc5695a58f9e" (UID: "40de722e-3cd0-4923-9aec-bc5695a58f9e"). InnerVolumeSpecName "kube-api-access-t5znx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:22.855698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:22.855663    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5znx\" (UniqueName: \"kubernetes.io/projected/40de722e-3cd0-4923-9aec-bc5695a58f9e-kube-api-access-t5znx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:23.546907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:23.546868    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"
Nov 03 11:15:23.546907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:23.546893    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b" event={"ID":"40de722e-3cd0-4923-9aec-bc5695a58f9e","Type":"ContainerDied","Data":"ae936f9dcfc936519a18d7128ede19c10d8b5f11e1ab94848ae514538fd8f932"}
Nov 03 11:15:23.547508 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:23.547001    5393 scope.go:122] "RemoveContainer" containerID="90db08a20d26c457e31cbf395c76ef5a931d2035225e23733a9533da552345d5"
Nov 03 11:15:23.573427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:23.573380    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"]
Nov 03 11:15:23.577814 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:23.577788    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cbeebd-29895075-mwm6b"]
Nov 03 11:15:24.234625 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:24.234593    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" path="/var/lib/kubelet/pods/1107f46f-c86c-4569-b755-9aa28e256af0/volumes"
Nov 03 11:15:24.235189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:24.235174    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" path="/var/lib/kubelet/pods/2534e7e5-ea82-432e-af50-cc00bb0fa611/volumes"
Nov 03 11:15:24.235610 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:24.235586    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" path="/var/lib/kubelet/pods/36bf3819-7aea-4e1c-ad10-66f59268dcf1/volumes"
Nov 03 11:15:24.236314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:24.236296    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" path="/var/lib/kubelet/pods/40de722e-3cd0-4923-9aec-bc5695a58f9e/volumes"
Nov 03 11:15:24.236715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:24.236700    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" path="/var/lib/kubelet/pods/4ccb486b-de82-43bc-b8fa-47e5a6a2bf81/volumes"
Nov 03 11:15:24.237154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:24.237139    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" path="/var/lib/kubelet/pods/763c6d00-52d4-4cc5-af13-1e7cfa1fc700/volumes"
Nov 03 11:15:24.237823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:24.237809    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" path="/var/lib/kubelet/pods/b5cea438-8f15-4414-8a1c-7f6df195d5a7/volumes"
Nov 03 11:15:24.238255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:24.238239    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" path="/var/lib/kubelet/pods/fb58eeb4-c1d2-449f-90d7-f92ab4eb1271/volumes"
Nov 03 11:15:30.396970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:30.396904    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:15:31.509259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:31.509225    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:15:43.226879 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:15:43.226838    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:15:43.445984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.445932    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"]
Nov 03 11:15:43.448287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448254    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.448287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448288    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448302    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448312    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448325    5393 cpu_manager.go:414] "containerMap: removing container" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448333    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448349    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448358    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448367    5393 cpu_manager.go:414] "containerMap: removing container" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448374    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448393    5393 cpu_manager.go:414] "containerMap: removing container" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448400    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448413    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448419    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448430    5393 cpu_manager.go:414] "containerMap: removing container" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448437    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448447    5393 cpu_manager.go:414] "containerMap: removing container" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448454    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448467    5393 cpu_manager.go:414] "containerMap: removing container" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:15:43.448477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448475    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448488    5393 cpu_manager.go:414] "containerMap: removing container" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448496    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448514    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448521    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448534    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448541    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448551    5393 cpu_manager.go:414] "containerMap: removing container" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448558    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448567    5393 cpu_manager.go:414] "containerMap: removing container" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448574    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448587    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448594    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448613    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448621    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448638    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448645    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448667    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448673    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448692    5393 cpu_manager.go:414] "containerMap: removing container" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448699    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448713    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448720    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448735    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448742    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448762    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448769    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448784    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448791    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448800    5393 cpu_manager.go:414] "containerMap: removing container" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448807    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448821    5393 cpu_manager.go:414] "containerMap: removing container" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448828    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448842    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448850    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448861    5393 cpu_manager.go:414] "containerMap: removing container" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448869    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448879    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448886    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448907    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448933    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448945    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448952    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448965    5393 cpu_manager.go:414] "containerMap: removing container" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448972    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448983    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.448991    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449007    5393 cpu_manager.go:414] "containerMap: removing container" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449013    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449024    5393 cpu_manager.go:414] "containerMap: removing container" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449030    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449227    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:15:43.449226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449245    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449256    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449267    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449277    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449287    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449299    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449308    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449316    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449324    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449331    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449339    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1107f46f-c86c-4569-b755-9aa28e256af0" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449350    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="51df457b-9f93-446b-8f3a-2d4dd62cfe92" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449361    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449372    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449380    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449388    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449398    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f191c99e-0c1a-4f16-9a57-231634a6cd6a" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449405    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449414    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449421    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449435    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449443    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449452    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449460    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449469    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449477    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449486    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449497    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449505    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449515    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449527    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449538    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449546    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449555    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.451103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.449563    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2534e7e5-ea82-432e-af50-cc00bb0fa611" containerName="cleanup"
Nov 03 11:15:43.460480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.460453    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464069    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-envoy\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464213    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmn7c\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-kube-api-access-hmn7c\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464256    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istiod-ca-cert\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464286    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-credential-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464409    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-data\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464437    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-certs\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464462    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-token\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464514    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-podinfo\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.464756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464732    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.465023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.464988    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"]
Nov 03 11:15:43.465182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.465159    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24376f-openshift-default-dockercfg-psf48\""
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.570819    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmn7c\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-kube-api-access-hmn7c\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.570894    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istiod-ca-cert\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.570952    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-credential-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.571117    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-data\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.571144    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-certs\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.571166    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-token\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.571223    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-podinfo\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.571316    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.572942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.571410    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-envoy\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.608950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.590795    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istiod-ca-cert\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.608950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.591102    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-credential-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.608950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.591340    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-data\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.608950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.591563    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-certs\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.608950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.593424    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-socket\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.639940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.631230    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-envoy\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.681829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.654508    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-token\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.681829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.680042    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-podinfo\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.682881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.682851    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmn7c\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-kube-api-access-hmn7c\") pod \"e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") " pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:43.780731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:43.780407    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:44.451981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:44.451616    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"]
Nov 03 11:15:44.464726 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:15:44.464099    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace83d08_f3da_4b9d_abfd_e8050fa828a5.slice/crio-4343cf6f31db35590d079bf8c2b9815e9f199e1f75b6cb6e79e1dfa25b43453d WatchSource:0}: Error finding container 4343cf6f31db35590d079bf8c2b9815e9f199e1f75b6cb6e79e1dfa25b43453d: Status 404 returned error can't find the container with id 4343cf6f31db35590d079bf8c2b9815e9f199e1f75b6cb6e79e1dfa25b43453d
Nov 03 11:15:44.475469 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:44.475420    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:44.475646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:44.475590    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:44.475725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:44.475694    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:15:45.130062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:45.130018    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" event={"ID":"ace83d08-f3da-4b9d-abfd-e8050fa828a5","Type":"ContainerStarted","Data":"1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f"}
Nov 03 11:15:45.130274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:45.130099    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" event={"ID":"ace83d08-f3da-4b9d-abfd-e8050fa828a5","Type":"ContainerStarted","Data":"4343cf6f31db35590d079bf8c2b9815e9f199e1f75b6cb6e79e1dfa25b43453d"}
Nov 03 11:15:45.153541 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:45.153485    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podStartSLOduration=2.1534712799999998 podStartE2EDuration="2.15347128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:15:45.152258791 +0000 UTC m=+2789.057113233" watchObservedRunningTime="2026-11-03 11:15:45.15347128 +0000 UTC m=+2789.058325720"
Nov 03 11:15:45.781156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:45.781116    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:45.788061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:45.788034    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:46.138863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:46.138833    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:46.139896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:46.139874    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:15:53.228943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:53.225904    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"]
Nov 03 11:15:53.874972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:53.874782    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"]
Nov 03 11:15:53.875188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:53.875050    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq" podUID="00388998-35db-47ae-91c5-0ddae7e97224" containerName="sim" containerID="cri-o://1e30d4171a2fb87f26bd6ba4fa150fa483fd3cfcca39aba3bac24eaa8e3859a2" gracePeriod=30
Nov 03 11:15:54.334809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.334758    5393 generic.go:356] "Generic (PLEG): container finished" podID="00388998-35db-47ae-91c5-0ddae7e97224" containerID="1e30d4171a2fb87f26bd6ba4fa150fa483fd3cfcca39aba3bac24eaa8e3859a2" exitCode=0
Nov 03 11:15:54.335348 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.334840    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq" event={"ID":"00388998-35db-47ae-91c5-0ddae7e97224","Type":"ContainerDied","Data":"1e30d4171a2fb87f26bd6ba4fa150fa483fd3cfcca39aba3bac24eaa8e3859a2"}
Nov 03 11:15:54.540773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.540748    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:15:54.649561 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.649522    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00388998-35db-47ae-91c5-0ddae7e97224-kube-api-access-k8ww9\" (UniqueName: \"kubernetes.io/projected/00388998-35db-47ae-91c5-0ddae7e97224-kube-api-access-k8ww9\") pod \"00388998-35db-47ae-91c5-0ddae7e97224\" (UID: \"00388998-35db-47ae-91c5-0ddae7e97224\") "
Nov 03 11:15:54.649752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.649611    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00388998-35db-47ae-91c5-0ddae7e97224-tls\" (UniqueName: \"kubernetes.io/secret/00388998-35db-47ae-91c5-0ddae7e97224-tls\") pod \"00388998-35db-47ae-91c5-0ddae7e97224\" (UID: \"00388998-35db-47ae-91c5-0ddae7e97224\") "
Nov 03 11:15:54.652040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.652010    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00388998-35db-47ae-91c5-0ddae7e97224-tls" pod "00388998-35db-47ae-91c5-0ddae7e97224" (UID: "00388998-35db-47ae-91c5-0ddae7e97224"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:15:54.652040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.652018    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00388998-35db-47ae-91c5-0ddae7e97224-kube-api-access-k8ww9" pod "00388998-35db-47ae-91c5-0ddae7e97224" (UID: "00388998-35db-47ae-91c5-0ddae7e97224"). InnerVolumeSpecName "kube-api-access-k8ww9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:54.761880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.760661    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ww9\" (UniqueName: \"kubernetes.io/projected/00388998-35db-47ae-91c5-0ddae7e97224-kube-api-access-k8ww9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:54.761880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:54.760703    5393 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/00388998-35db-47ae-91c5-0ddae7e97224-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:55.353945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:55.353736    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq" event={"ID":"00388998-35db-47ae-91c5-0ddae7e97224","Type":"ContainerDied","Data":"e7529d76557da992c6116f1ef0e2f5fdb16b6f236e5d2e5bd2127e1fe23b4318"}
Nov 03 11:15:55.353945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:55.353800    5393 scope.go:122] "RemoveContainer" containerID="1e30d4171a2fb87f26bd6ba4fa150fa483fd3cfcca39aba3bac24eaa8e3859a2"
Nov 03 11:15:55.353945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:55.353951    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"
Nov 03 11:15:55.388704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:55.388640    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"]
Nov 03 11:15:55.395161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:55.395102    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-2c5853/e2e-usage-sim-cba7ed-59647847cb-dzwgq"]
Nov 03 11:15:56.234661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:56.234628    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00388998-35db-47ae-91c5-0ddae7e97224" path="/var/lib/kubelet/pods/00388998-35db-47ae-91c5-0ddae7e97224/volumes"
Nov 03 11:15:58.227557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.227524    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w" podUID="5a2f2b77-7d92-47c1-aa7d-da5596769d13" containerName="payload-processing" containerID="cri-o://64ad7b5e54a5335c7d26e3426c8318800b0453f208ef16c1f3e1dad2197990ce" gracePeriod=25
Nov 03 11:15:58.467528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.467489    5393 generic.go:356] "Generic (PLEG): container finished" podID="5a2f2b77-7d92-47c1-aa7d-da5596769d13" containerID="64ad7b5e54a5335c7d26e3426c8318800b0453f208ef16c1f3e1dad2197990ce" exitCode=0
Nov 03 11:15:58.467722 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.467557    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w" event={"ID":"5a2f2b77-7d92-47c1-aa7d-da5596769d13","Type":"ContainerDied","Data":"64ad7b5e54a5335c7d26e3426c8318800b0453f208ef16c1f3e1dad2197990ce"}
Nov 03 11:15:58.616807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.616768    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:15:58.775837 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.775797    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-plugins-config-volume\") pod \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") "
Nov 03 11:15:58.776138 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.776117    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-provider-credentials\") pod \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") "
Nov 03 11:15:58.776257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.776241    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-routing-overlay-volume\") pod \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") "
Nov 03 11:15:58.776343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.776315    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-plugins-config-volume" pod "5a2f2b77-7d92-47c1-aa7d-da5596769d13" (UID: "5a2f2b77-7d92-47c1-aa7d-da5596769d13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:58.776419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.776345    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-kube-api-access-47gfm\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-kube-api-access-47gfm\") pod \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\" (UID: \"5a2f2b77-7d92-47c1-aa7d-da5596769d13\") "
Nov 03 11:15:58.776660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.776637    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-routing-overlay-volume" pod "5a2f2b77-7d92-47c1-aa7d-da5596769d13" (UID: "5a2f2b77-7d92-47c1-aa7d-da5596769d13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:15:58.777254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.777178    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:58.777254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.777206    5393 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a2f2b77-7d92-47c1-aa7d-da5596769d13-routing-overlay-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:58.778805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.778784    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-provider-credentials" pod "5a2f2b77-7d92-47c1-aa7d-da5596769d13" (UID: "5a2f2b77-7d92-47c1-aa7d-da5596769d13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:58.779210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.779190    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-kube-api-access-47gfm" pod "5a2f2b77-7d92-47c1-aa7d-da5596769d13" (UID: "5a2f2b77-7d92-47c1-aa7d-da5596769d13"). InnerVolumeSpecName "kube-api-access-47gfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:15:58.878860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.878806    5393 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-provider-credentials\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:58.878860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:58.878859    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47gfm\" (UniqueName: \"kubernetes.io/projected/5a2f2b77-7d92-47c1-aa7d-da5596769d13-kube-api-access-47gfm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:15:59.653092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:59.647858    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w" event={"ID":"5a2f2b77-7d92-47c1-aa7d-da5596769d13","Type":"ContainerDied","Data":"b7c5661bb80899ac0db5ca8053796601ccd7f81ce4e81ce9eaa951e46164e5c8"}
Nov 03 11:15:59.653092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:59.647962    5393 scope.go:122] "RemoveContainer" containerID="64ad7b5e54a5335c7d26e3426c8318800b0453f208ef16c1f3e1dad2197990ce"
Nov 03 11:15:59.653092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:59.648129    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"
Nov 03 11:15:59.850174 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:59.850122    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"]
Nov 03 11:15:59.856351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:15:59.856324    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cbeebd/payload-processing-external-model-e2e-extmod-a-cbeebd-57fd8xd7w"]
Nov 03 11:16:00.266531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:00.266491    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2f2b77-7d92-47c1-aa7d-da5596769d13" path="/var/lib/kubelet/pods/5a2f2b77-7d92-47c1-aa7d-da5596769d13/volumes"
Nov 03 11:16:02.833254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.833214    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"]
Nov 03 11:16:02.834629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834600    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5a2f2b77-7d92-47c1-aa7d-da5596769d13" containerName="payload-processing"
Nov 03 11:16:02.834629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834624    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2f2b77-7d92-47c1-aa7d-da5596769d13" containerName="payload-processing"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834639    5393 cpu_manager.go:414] "containerMap: removing container" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834646    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834662    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834671    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e754a45-81bc-45e1-9949-5c8c9f6a638c" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834681    5393 cpu_manager.go:414] "containerMap: removing container" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834688    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb58eeb4-c1d2-449f-90d7-f92ab4eb1271" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834707    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834714    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb3f8d3-540f-4c0d-b1ee-525b4e4d22d5" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834725    5393 cpu_manager.go:414] "containerMap: removing container" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834731    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bf3819-7aea-4e1c-ad10-66f59268dcf1" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834747    5393 cpu_manager.go:414] "containerMap: removing container" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834753    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834775    5393 cpu_manager.go:414] "containerMap: removing container" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834781    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834801    5393 cpu_manager.go:414] "containerMap: removing container" podUID="00388998-35db-47ae-91c5-0ddae7e97224" containerName="sim"
Nov 03 11:16:02.834812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.834807    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00388998-35db-47ae-91c5-0ddae7e97224" containerName="sim"
Nov 03 11:16:02.835646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.835006    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="26324c17-9964-4c55-8f6c-30c6511f7048" containerName="cleanup"
Nov 03 11:16:02.835646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.835019    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bea47f-c0e1-4a14-acd0-920b5b43e4b3" containerName="cleanup"
Nov 03 11:16:02.835646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.835035    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2f2b77-7d92-47c1-aa7d-da5596769d13" containerName="payload-processing"
Nov 03 11:16:02.835646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.835045    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="763c6d00-52d4-4cc5-af13-1e7cfa1fc700" containerName="cleanup"
Nov 03 11:16:02.835646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.835057    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="40de722e-3cd0-4923-9aec-bc5695a58f9e" containerName="cleanup"
Nov 03 11:16:02.835646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.835070    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccb486b-de82-43bc-b8fa-47e5a6a2bf81" containerName="cleanup"
Nov 03 11:16:02.835646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.835079    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="00388998-35db-47ae-91c5-0ddae7e97224" containerName="sim"
Nov 03 11:16:02.835646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.835089    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cea438-8f15-4414-8a1c-7f6df195d5a7" containerName="cleanup"
Nov 03 11:16:02.839837 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.839799    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.852499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.851385    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"]
Nov 03 11:16:02.919075 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919039    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.919075 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919076    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.919321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919289    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.919417 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919399    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.919496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919466    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.919535 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919512    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.919590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919573    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.919700 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919686    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7mc\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-kube-api-access-sj7mc\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:02.919746 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:02.919715    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7086125d-88b6-4b83-8e78-9cc56eadec7e-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.021532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.021492    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7mc\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-kube-api-access-sj7mc\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.021713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.021564    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7086125d-88b6-4b83-8e78-9cc56eadec7e-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.021784 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.021755    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.021835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.021785    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.021950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.021934    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022015    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022078    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022127    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022335    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-credential-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022336    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022424    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7086125d-88b6-4b83-8e78-9cc56eadec7e-istiod-ca-cert\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022244    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-data\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022637 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022489    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-socket\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.022637 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.022599    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-certs\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.025482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.025454    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-podinfo\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.029198 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.029170    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-envoy\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.032777 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.032750    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-token\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.032894 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.032864    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7mc\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-kube-api-access-sj7mc\") pod \"e2e-inf-a-055d82-openshift-default-79544dc787-jbsql\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") " pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.165293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.165251    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:03.511965 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:16:03.511911    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7086125d_88b6_4b83_8e78_9cc56eadec7e.slice/crio-23ebc60c637ac5a85e18f1f0df159a6e68174e745a0c24eccaa27de51631338c WatchSource:0}: Error finding container 23ebc60c637ac5a85e18f1f0df159a6e68174e745a0c24eccaa27de51631338c: Status 404 returned error can't find the container with id 23ebc60c637ac5a85e18f1f0df159a6e68174e745a0c24eccaa27de51631338c
Nov 03 11:16:03.513542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.513516    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"]
Nov 03 11:16:03.515350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.515318    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.515443 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.515394    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.515443 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.515436    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:03.849079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.849008    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" event={"ID":"7086125d-88b6-4b83-8e78-9cc56eadec7e","Type":"ContainerStarted","Data":"ec512e80510fbe3e08156377a6d1eea55ad535ddfdef0a11627b5a870d1514a1"}
Nov 03 11:16:03.849489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.849109    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" event={"ID":"7086125d-88b6-4b83-8e78-9cc56eadec7e","Type":"ContainerStarted","Data":"23ebc60c637ac5a85e18f1f0df159a6e68174e745a0c24eccaa27de51631338c"}
Nov 03 11:16:03.869425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:03.869361    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podStartSLOduration=1.869342508 podStartE2EDuration="1.869342508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:03.867124059 +0000 UTC m=+2807.771978521" watchObservedRunningTime="2026-11-03 11:16:03.869342508 +0000 UTC m=+2807.774196950"
Nov 03 11:16:04.166190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:04.166104    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:04.167763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:04.167733    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:16:04.167960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:04.167794    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 11:16:05.193114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:05.192565    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:16:05.193114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:05.192655    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 11:16:06.173112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:06.171965    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:16:06.173112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:06.172040    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 11:16:06.511185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:06.511088    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:16:06.511185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:06.511160    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:16:07.166589 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.166530    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:16:07.166816 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.166593    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 11:16:07.895959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.880794    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"]
Nov 03 11:16:07.904243 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.904212    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:07.933910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.928244    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a68103\"/\"kube-root-ca.crt\""
Nov 03 11:16:07.933910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.928355    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a68103\"/\"default-dockercfg-t8hbr\""
Nov 03 11:16:07.933910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.928424    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a68103\"/\"test-model-a68103-kserve-self-signed-certs\""
Nov 03 11:16:07.933910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.928531    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a68103\"/\"openshift-service-ca.crt\""
Nov 03 11:16:07.933910 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:07.933663    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"]
Nov 03 11:16:08.044886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.044841    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-dshm\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.058062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.048932    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-home\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.058062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.048977    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhtn\" (UniqueName: \"kubernetes.io/projected/feaf2442-f46e-4d47-8b95-6b5465574620-kube-api-access-xvhtn\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.058062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.049065    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-tmp-dir\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.058062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.049104    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feaf2442-f46e-4d47-8b95-6b5465574620-tls-certs\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.058062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.049162    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-model-cache\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.157642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.156137    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-dshm\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.157642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.156445    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-home\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.157642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.156476    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhtn\" (UniqueName: \"kubernetes.io/projected/feaf2442-f46e-4d47-8b95-6b5465574620-kube-api-access-xvhtn\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.157642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.156551    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-tmp-dir\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.157642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.156576    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feaf2442-f46e-4d47-8b95-6b5465574620-tls-certs\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.157642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.156625    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-model-cache\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.157642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.157267    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-model-cache\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.184790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.183372    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:16:08.184790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.183448    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 11:16:08.184790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.184168    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-tmp-dir\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.184790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.184419    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-home\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.197991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.193186    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-dshm\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.228960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.227948    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feaf2442-f46e-4d47-8b95-6b5465574620-tls-certs\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.237970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.229622    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhtn\" (UniqueName: \"kubernetes.io/projected/feaf2442-f46e-4d47-8b95-6b5465574620-kube-api-access-xvhtn\") pod \"test-model-a68103-kserve-6f859fbd9c-gdx98\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") " pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.279961 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.276587    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:08.736091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:08.736061    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"]
Nov 03 11:16:09.077779 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.077689    5393 generic.go:356] "Generic (PLEG): container finished" podID="2310479c-98fb-4cc8-8b03-59003bd96943" containerID="64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d" exitCode=0
Nov 03 11:16:09.078211 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.077766    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" event={"ID":"2310479c-98fb-4cc8-8b03-59003bd96943","Type":"ContainerDied","Data":"64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d"}
Nov 03 11:16:09.078320 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.078302    5393 scope.go:122] "RemoveContainer" containerID="64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d"
Nov 03 11:16:09.079997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.079970    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98" event={"ID":"feaf2442-f46e-4d47-8b95-6b5465574620","Type":"ContainerStarted","Data":"b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605"}
Nov 03 11:16:09.080133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.080011    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98" event={"ID":"feaf2442-f46e-4d47-8b95-6b5465574620","Type":"ContainerStarted","Data":"f171136623e658c022dd515bff50004a07f3445493b65d86f4dfe13763c52faa"}
Nov 03 11:16:09.080653 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.080627    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:09.080743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.080693    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:09.080743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.080727    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:09.131959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.131881    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98" podStartSLOduration=2.131862037 podStartE2EDuration="2.131862037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:09.127969859 +0000 UTC m=+2813.032824301" watchObservedRunningTime="2026-11-03 11:16:09.131862037 +0000 UTC m=+2813.036716479"
Nov 03 11:16:09.166713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.166649    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:16:09.166908 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.166736    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 11:16:09.293406 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:09.289037    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:10.102831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:10.099548    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" event={"ID":"2310479c-98fb-4cc8-8b03-59003bd96943","Type":"ContainerStarted","Data":"ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f"}
Nov 03 11:16:10.166962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:10.166715    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 11:16:10.166962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:10.166773    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 11:16:10.276819 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:10.276784    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:16:10.277218 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:10.277192    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body=
Nov 03 11:16:10.277312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:10.277247    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused"
Nov 03 11:16:11.171842 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:11.171807    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:11.172383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:11.172331    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:11.173110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:11.173090    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:16:11.276652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:11.276491    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body=
Nov 03 11:16:11.276652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:11.276557    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused"
Nov 03 11:16:12.276182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:12.276139    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body=
Nov 03 11:16:12.276682 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:12.276209    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused"
Nov 03 11:16:13.215316 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:16:13.215275    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:16:13.276442 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:13.276403    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body=
Nov 03 11:16:13.277019 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:13.276467    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused"
Nov 03 11:16:14.280622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:14.280112    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body=
Nov 03 11:16:14.280622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:14.280186    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused"
Nov 03 11:16:15.276071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:15.276035    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body=
Nov 03 11:16:15.276282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:15.276128    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused"
Nov 03 11:16:16.276496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.276460    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body=
Nov 03 11:16:16.276975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.276516    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused"
Nov 03 11:16:16.718277 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.718239    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"]
Nov 03 11:16:16.723595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.723569    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:16.733447 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.733414    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-68aa46\""
Nov 03 11:16:16.735540 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.735511    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"]
Nov 03 11:16:16.742336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.742311    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"]
Nov 03 11:16:16.751407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.751381    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:16.761485 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.761451    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"]
Nov 03 11:16:16.764398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.764371    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-68aa46-dockercfg-hbg79\""
Nov 03 11:16:16.780822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.780793    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fffc3d5-3d17-471e-8906-5a94ed21d14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9\" (UID: \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:16.781183 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.781158    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcrxp\" (UniqueName: \"kubernetes.io/projected/9fffc3d5-3d17-471e-8906-5a94ed21d14c-kube-api-access-xcrxp\") pod \"payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9\" (UID: \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:16.781327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.781308    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbmt\" (UniqueName: \"kubernetes.io/projected/c6583710-32ad-4a57-9935-cb8953eaea21-kube-api-access-wbbmt\") pod \"payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k\" (UID: \"c6583710-32ad-4a57-9935-cb8953eaea21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:16.781427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.781403    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6583710-32ad-4a57-9935-cb8953eaea21-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k\" (UID: \"c6583710-32ad-4a57-9935-cb8953eaea21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:16.917940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.917400    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcrxp\" (UniqueName: \"kubernetes.io/projected/9fffc3d5-3d17-471e-8906-5a94ed21d14c-kube-api-access-xcrxp\") pod \"payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9\" (UID: \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:16.917940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.917560    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbmt\" (UniqueName: \"kubernetes.io/projected/c6583710-32ad-4a57-9935-cb8953eaea21-kube-api-access-wbbmt\") pod \"payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k\" (UID: \"c6583710-32ad-4a57-9935-cb8953eaea21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:16.917940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.917635    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6583710-32ad-4a57-9935-cb8953eaea21-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k\" (UID: \"c6583710-32ad-4a57-9935-cb8953eaea21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:16.917940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.917752    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fffc3d5-3d17-471e-8906-5a94ed21d14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9\" (UID: \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:16.929942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.918814    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fffc3d5-3d17-471e-8906-5a94ed21d14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9\" (UID: \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:16.929942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.920158    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6583710-32ad-4a57-9935-cb8953eaea21-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k\" (UID: \"c6583710-32ad-4a57-9935-cb8953eaea21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:16.958948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.958861    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcrxp\" (UniqueName: \"kubernetes.io/projected/9fffc3d5-3d17-471e-8906-5a94ed21d14c-kube-api-access-xcrxp\") pod \"payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9\" (UID: \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:16.985989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:16.973536    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbmt\" (UniqueName: \"kubernetes.io/projected/c6583710-32ad-4a57-9935-cb8953eaea21-kube-api-access-wbbmt\") pod \"payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k\" (UID: \"c6583710-32ad-4a57-9935-cb8953eaea21\") " pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:17.054941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:17.051322    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:17.129511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:17.112334    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:17.341993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:17.333295    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:16:17.344932 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:17.344102    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:16:17.346899 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:17.346719    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:16:17.631500 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:16:17.631432    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6583710_32ad_4a57_9935_cb8953eaea21.slice/crio-150add15c9e5a3df5de34865dd6e74389c5ac620656d3b7819b534a15e526306 WatchSource:0}: Error finding container 150add15c9e5a3df5de34865dd6e74389c5ac620656d3b7819b534a15e526306: Status 404 returned error can't find the container with id 150add15c9e5a3df5de34865dd6e74389c5ac620656d3b7819b534a15e526306
Nov 03 11:16:17.643008 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:17.642982    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"]
Nov 03 11:16:17.655063 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:16:17.655030    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fffc3d5_3d17_471e_8906_5a94ed21d14c.slice/crio-e9e53af48ecfb1c43978fa81741c008d8921c535894e62ac9d1245edbafeed84 WatchSource:0}: Error finding container e9e53af48ecfb1c43978fa81741c008d8921c535894e62ac9d1245edbafeed84: Status 404 returned error can't find the container with id e9e53af48ecfb1c43978fa81741c008d8921c535894e62ac9d1245edbafeed84
Nov 03 11:16:17.655241 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:17.655104    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"]
Nov 03 11:16:18.278626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.278597    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:18.291833 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.291803    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:18.355263 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.355225    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k" event={"ID":"c6583710-32ad-4a57-9935-cb8953eaea21","Type":"ContainerStarted","Data":"40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae"}
Nov 03 11:16:18.355755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.355288    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k" event={"ID":"c6583710-32ad-4a57-9935-cb8953eaea21","Type":"ContainerStarted","Data":"150add15c9e5a3df5de34865dd6e74389c5ac620656d3b7819b534a15e526306"}
Nov 03 11:16:18.355755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.355316    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:18.357288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.357258    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9" event={"ID":"9fffc3d5-3d17-471e-8906-5a94ed21d14c","Type":"ContainerStarted","Data":"d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269"}
Nov 03 11:16:18.357437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.357306    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9" event={"ID":"9fffc3d5-3d17-471e-8906-5a94ed21d14c","Type":"ContainerStarted","Data":"e9e53af48ecfb1c43978fa81741c008d8921c535894e62ac9d1245edbafeed84"}
Nov 03 11:16:18.357557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.357539    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:18.372284 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.369649    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:16:18.376607 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:18.376564    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k" podStartSLOduration=2.376552412 podStartE2EDuration="2.376552412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:18.372568942 +0000 UTC m=+2822.277423384" watchObservedRunningTime="2026-11-03 11:16:18.376552412 +0000 UTC m=+2822.281406853"
Nov 03 11:16:19.013715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.013223    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9" podStartSLOduration=3.013201093 podStartE2EDuration="3.013201093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:18.416155344 +0000 UTC m=+2822.321009786" watchObservedRunningTime="2026-11-03 11:16:19.013201093 +0000 UTC m=+2822.918055535"
Nov 03 11:16:19.018393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.017264    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"]
Nov 03 11:16:19.031365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.031335    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"]
Nov 03 11:16:19.031560 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.031450    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.035082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.035056    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3eb43f\"/\"default-dockercfg-55dz2\""
Nov 03 11:16:19.039396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.039367    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3eb43f\"/\"test-model-3eb43f-kserve-self-signed-certs\""
Nov 03 11:16:19.043282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.040544    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3eb43f\"/\"openshift-service-ca.crt\""
Nov 03 11:16:19.043425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.040695    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3eb43f\"/\"kube-root-ca.crt\""
Nov 03 11:16:19.092653 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.092618    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-model-cache\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.092653 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.092664    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tls-certs\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.092987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.092690    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tmp-dir\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.092987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.092875    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lffgz\" (UniqueName: \"kubernetes.io/projected/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-kube-api-access-lffgz\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.093241 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.093213    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-dshm\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.093311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.093263    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-home\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.199944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.198459    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-dshm\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.199944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.198521    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-home\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.199944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.198561    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-model-cache\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.199944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.198588    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tls-certs\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.199944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.198612    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tmp-dir\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.199944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.198692    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffgz\" (UniqueName: \"kubernetes.io/projected/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-kube-api-access-lffgz\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.208942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.205964    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-model-cache\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.208942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.206229    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-home\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.247947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.240081    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tls-certs\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.252418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.250941    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lffgz\" (UniqueName: \"kubernetes.io/projected/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-kube-api-access-lffgz\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.252418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.251661    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tmp-dir\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.284008 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.280216    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-dshm\") pod \"test-model-3eb43f-kserve-59d588c649-269d6\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") " pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.376943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.370684    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:19.747688 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:16:19.747654    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb974d9d2_1bb1_41ad_bc3c_872ae1e2c057.slice/crio-4c7dda5ada8604becfbddb558a8f27031d59dc2711c4e356704f917597de554d WatchSource:0}: Error finding container 4c7dda5ada8604becfbddb558a8f27031d59dc2711c4e356704f917597de554d: Status 404 returned error can't find the container with id 4c7dda5ada8604becfbddb558a8f27031d59dc2711c4e356704f917597de554d
Nov 03 11:16:19.748379 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:19.748286    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"]
Nov 03 11:16:20.407899 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:20.407858    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6" event={"ID":"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057","Type":"ContainerStarted","Data":"69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c"}
Nov 03 11:16:20.408365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:20.407952    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6" event={"ID":"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057","Type":"ContainerStarted","Data":"4c7dda5ada8604becfbddb558a8f27031d59dc2711c4e356704f917597de554d"}
Nov 03 11:16:20.427584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:20.427525    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6" podStartSLOduration=2.42751097 podStartE2EDuration="2.42751097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:20.424683448 +0000 UTC m=+2824.329537890" watchObservedRunningTime="2026-11-03 11:16:20.42751097 +0000 UTC m=+2824.332365433"
Nov 03 11:16:21.371555 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:21.371523    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:29.372039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:29.371943    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:29.385569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:29.385537    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:29.401618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:29.401589    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:16:29.435219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:29.435184    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:16:29.778642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:29.778556    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:16:43.214282 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:16:43.214228    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:16:45.239425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.239383    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"]
Nov 03 11:16:45.254728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.254034    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.258210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.258183    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"]
Nov 03 11:16:45.260481 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.260452    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-055d82\"/\"default-dockercfg-kzwq5\""
Nov 03 11:16:45.260625 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.260569    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-055d82\"/\"kube-root-ca.crt\""
Nov 03 11:16:45.261244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.261220    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-055d82\"/\"openshift-service-ca.crt\""
Nov 03 11:16:45.261542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.261500    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-055d82\"/\"test-model-055d82-kserve-self-signed-certs\""
Nov 03 11:16:45.397468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.397425    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-home\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.397894 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.397854    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-tmp-dir\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.398212 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.398191    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a21254-4030-4431-8334-4ac98d53c6ab-tls-certs\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.398426 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.398407    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-model-cache\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.398839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.398818    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-dshm\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.399131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.399109    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmx82\" (UniqueName: \"kubernetes.io/projected/c7a21254-4030-4431-8334-4ac98d53c6ab-kube-api-access-pmx82\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.501447 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.501350    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-dshm\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.502347 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.502306    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmx82\" (UniqueName: \"kubernetes.io/projected/c7a21254-4030-4431-8334-4ac98d53c6ab-kube-api-access-pmx82\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.506178 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.506147    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-home\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.530945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.529146    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-tmp-dir\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.530945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.529284    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a21254-4030-4431-8334-4ac98d53c6ab-tls-certs\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.530945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.529324    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-model-cache\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.530945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.530059    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-model-cache\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.530945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.506653    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-home\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.530945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.530327    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-tmp-dir\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.530945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.507301    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-dshm\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.549026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.548994    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a21254-4030-4431-8334-4ac98d53c6ab-tls-certs\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.562230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.551301    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmx82\" (UniqueName: \"kubernetes.io/projected/c7a21254-4030-4431-8334-4ac98d53c6ab-kube-api-access-pmx82\") pod \"test-model-055d82-kserve-8568995767-v5zxq\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") " pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:45.583276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:45.583242    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:46.162262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:46.162228    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"]
Nov 03 11:16:46.277783 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:16:46.277723    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7a21254_4030_4431_8334_4ac98d53c6ab.slice/crio-df0ce1558003f26fc392d15e2f03e898e9b01ce1c158c6f4fd519975976c6a0c WatchSource:0}: Error finding container df0ce1558003f26fc392d15e2f03e898e9b01ce1c158c6f4fd519975976c6a0c: Status 404 returned error can't find the container with id df0ce1558003f26fc392d15e2f03e898e9b01ce1c158c6f4fd519975976c6a0c
Nov 03 11:16:47.299951 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:16:47.295458    5393 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.056s"
Nov 03 11:16:47.314518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:47.314480    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq" event={"ID":"c7a21254-4030-4431-8334-4ac98d53c6ab","Type":"ContainerStarted","Data":"df0ce1558003f26fc392d15e2f03e898e9b01ce1c158c6f4fd519975976c6a0c"}
Nov 03 11:16:48.324407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:48.324357    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq" event={"ID":"c7a21254-4030-4431-8334-4ac98d53c6ab","Type":"ContainerStarted","Data":"13912600d24687c351e91b0dc40cf371fd25f2188dc6f82e11a29197c286b1c1"}
Nov 03 11:16:48.344159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:48.344096    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq" podStartSLOduration=3.344077224 podStartE2EDuration="3.344077224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:16:48.342128829 +0000 UTC m=+2852.246983270" watchObservedRunningTime="2026-11-03 11:16:48.344077224 +0000 UTC m=+2852.248931666"
Nov 03 11:16:48.585265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:48.585171    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:54.436664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:54.436625    5393 generic.go:356] "Generic (PLEG): container finished" podID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerID="1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f" exitCode=0
Nov 03 11:16:54.437657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:54.437367    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" event={"ID":"ace83d08-f3da-4b9d-abfd-e8050fa828a5","Type":"ContainerDied","Data":"1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f"}
Nov 03 11:16:54.439007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:54.438987    5393 scope.go:122] "RemoveContainer" containerID="1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f"
Nov 03 11:16:54.446698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:54.446651    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:54.446842 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:54.446763    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:54.446842 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:54.446813    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:16:55.560155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:55.558423    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" event={"ID":"ace83d08-f3da-4b9d-abfd-e8050fa828a5","Type":"ContainerStarted","Data":"f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a"}
Nov 03 11:16:55.610942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:55.595625    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:55.671911 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:55.671878    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:55.787948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:55.783581    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:16:55.809943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:55.806286    5393 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 03 11:16:55.809943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:55.806353    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 03 11:16:56.579306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:56.579274    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:16:56.788320 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:56.788278    5393 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 03 11:16:56.788649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:56.788620    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 03 11:16:57.794947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:57.791135    5393 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 03 11:16:57.794947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:57.791234    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 03 11:16:58.782604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:58.782568    5393 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 03 11:16:58.782898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:58.782867    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 03 11:16:59.940946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:59.926414    5393 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 03 11:16:59.982528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:59.982484    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 03 11:17:00.006407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:16:59.982659    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:17:00.177450 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.177418    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"]
Nov 03 11:17:00.190264 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.190128    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.207076 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.197205    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24376f\"/\"test-model-24376f-kserve-self-signed-certs\""
Nov 03 11:17:00.209529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.209497    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24376f\"/\"openshift-service-ca.crt\""
Nov 03 11:17:00.209664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.209640    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24376f\"/\"default-dockercfg-sjnq7\""
Nov 03 11:17:00.210184 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.209906    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24376f\"/\"kube-root-ca.crt\""
Nov 03 11:17:00.226816 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.226784    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"]
Nov 03 11:17:00.228951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.228901    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-home\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.229101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.229027    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-model-cache\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.229501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.229106    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-tmp-dir\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.229501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.229149    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791f8324-baed-4007-975c-9995b66647f4-tls-certs\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.229501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.229261    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg7z\" (UniqueName: \"kubernetes.io/projected/791f8324-baed-4007-975c-9995b66647f4-kube-api-access-bsg7z\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.229501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.229303    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-dshm\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.330365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.330326    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg7z\" (UniqueName: \"kubernetes.io/projected/791f8324-baed-4007-975c-9995b66647f4-kube-api-access-bsg7z\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.330690 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.330664    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-dshm\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.330827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.330766    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-home\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.330902 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.330852    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-model-cache\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.331123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.330960    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-tmp-dir\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.331123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.331006    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791f8324-baed-4007-975c-9995b66647f4-tls-certs\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.331667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.331644    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-home\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.331796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.331703    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-model-cache\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.331995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.331972    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-tmp-dir\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.333532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.333505    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-dshm\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.333949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.333895    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791f8324-baed-4007-975c-9995b66647f4-tls-certs\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.338439 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.338413    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg7z\" (UniqueName: \"kubernetes.io/projected/791f8324-baed-4007-975c-9995b66647f4-kube-api-access-bsg7z\") pod \"test-model-24376f-kserve-6448887f87-mn8lw\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") " pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.527668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.527570    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:00.816180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.816080    5393 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 03 11:17:00.816180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:00.816145    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 03 11:17:01.059536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:01.059385    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"]
Nov 03 11:17:01.060414 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:17:01.060387    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod791f8324_baed_4007_975c_9995b66647f4.slice/crio-43a3601e6d00a7c7925beef060946e2a53e00729eb7167549008d061f5fcc31d WatchSource:0}: Error finding container 43a3601e6d00a7c7925beef060946e2a53e00729eb7167549008d061f5fcc31d: Status 404 returned error can't find the container with id 43a3601e6d00a7c7925beef060946e2a53e00729eb7167549008d061f5fcc31d
Nov 03 11:17:01.139009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:01.138972    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:17:01.162532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:01.162504    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw" event={"ID":"791f8324-baed-4007-975c-9995b66647f4","Type":"ContainerStarted","Data":"43a3601e6d00a7c7925beef060946e2a53e00729eb7167549008d061f5fcc31d"}
Nov 03 11:17:01.378711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:01.378662    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal" podUID="3041328657ebd4fdb2a0f070aa3e29b6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Nov 03 11:17:01.819049 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:01.818948    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:17:01.821391 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:01.821346    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:17:02.172411 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:02.172367    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw" event={"ID":"791f8324-baed-4007-975c-9995b66647f4","Type":"ContainerStarted","Data":"6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c"}
Nov 03 11:17:02.190512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:02.190459    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw" podStartSLOduration=3.190443783 podStartE2EDuration="3.190443783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:02.187974957 +0000 UTC m=+2866.092829399" watchObservedRunningTime="2026-11-03 11:17:02.190443783 +0000 UTC m=+2866.095298224"
Nov 03 11:17:02.528314 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:02.528236    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:10.527796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:10.527756    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:10.541332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:10.541303    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:11.415650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:11.415616    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:17:13.192365 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:17:13.192325    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:17:16.193999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.193964    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"]
Nov 03 11:17:16.199161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.199137    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.207515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.207457    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a68103-openshift-default-dockercfg-t5bkr\""
Nov 03 11:17:16.212092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.209201    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"]
Nov 03 11:17:16.312513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.312485    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-data\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.312726 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.312677    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istiod-ca-cert\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.312726 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.312720    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-certs\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.312832 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.312812    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-envoy\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.312883 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.312870    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-token\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.313000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.312980    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxw6\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-kube-api-access-dtxw6\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.313072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.313013    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.313072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.313030    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-podinfo\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.313206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.313180    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-credential-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.415375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415339    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istiod-ca-cert\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.415569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415398    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-certs\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.415569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415465    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-envoy\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.415569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415521    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-token\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.415780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415571    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxw6\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-kube-api-access-dtxw6\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.415780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415599    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.415780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415628    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-podinfo\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.415780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415691    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-credential-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.416116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.415984    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-certs\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.416116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.416052    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-data\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.416244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.416203    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istiod-ca-cert\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.416308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.416282    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.416712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.416686    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-data\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.417086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.417061    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-credential-socket\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.419204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.419179    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-envoy\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.419350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.419329    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-podinfo\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.433229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.433129    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-token\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.433356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.433251    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxw6\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-kube-api-access-dtxw6\") pod \"e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") " pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:16.550750 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.550674    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a68103-openshift-default-dockercfg-t5bkr\""
Nov 03 11:17:16.558972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:16.558940    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:17.217949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:17.217904    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"]
Nov 03 11:17:17.218307 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:17:17.218242    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccacc68_1e07_47c4_9878_6bf1dcafbd5b.slice/crio-3cf5832f806333bbb9ed8b0ba7917f3c34d281e8f4e97729ec2742b67e9f16b8 WatchSource:0}: Error finding container 3cf5832f806333bbb9ed8b0ba7917f3c34d281e8f4e97729ec2742b67e9f16b8: Status 404 returned error can't find the container with id 3cf5832f806333bbb9ed8b0ba7917f3c34d281e8f4e97729ec2742b67e9f16b8
Nov 03 11:17:17.221596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:17.221564    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:17.221689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:17.221630    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:17.221689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:17.221663    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:17:17.537967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:17.537859    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" event={"ID":"fccacc68-1e07-47c4-9878-6bf1dcafbd5b","Type":"ContainerStarted","Data":"6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96"}
Nov 03 11:17:17.538145 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:17.537966    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" event={"ID":"fccacc68-1e07-47c4-9878-6bf1dcafbd5b","Type":"ContainerStarted","Data":"3cf5832f806333bbb9ed8b0ba7917f3c34d281e8f4e97729ec2742b67e9f16b8"}
Nov 03 11:17:17.559311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:17.559278    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:17.563689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:17.563623    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podStartSLOduration=1.5636063949999999 podStartE2EDuration="1.563606395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:17.55705316 +0000 UTC m=+2881.461907604" watchObservedRunningTime="2026-11-03 11:17:17.563606395 +0000 UTC m=+2881.468460838"
Nov 03 11:17:18.561346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:18.561315    5393 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body=
Nov 03 11:17:18.561771 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:18.561367    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused"
Nov 03 11:17:19.559559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:19.559528    5393 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body=
Nov 03 11:17:19.559769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:19.559595    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused"
Nov 03 11:17:20.592943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:20.578039    5393 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body=
Nov 03 11:17:20.592943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:20.578116    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused"
Nov 03 11:17:21.560292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:21.560254    5393 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body=
Nov 03 11:17:21.560514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:21.560329    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused"
Nov 03 11:17:22.559385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:22.559349    5393 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body=
Nov 03 11:17:22.559793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:22.559399    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused"
Nov 03 11:17:23.559766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:23.559731    5393 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body=
Nov 03 11:17:23.560303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:23.559797    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused"
Nov 03 11:17:24.559449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:24.559405    5393 patch_prober.go:29] interesting pod/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body=
Nov 03 11:17:24.559633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:24.559480    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused"
Nov 03 11:17:25.564485 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:25.564454    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:25.565009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:25.564986    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:25.566329 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:25.566290    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:17:42.747110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.747069    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"]
Nov 03 11:17:42.753703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.753675    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:42.766715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.766673    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"]
Nov 03 11:17:42.768849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.768822    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c0a748-serving-cert\""
Nov 03 11:17:42.871046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.871003    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:42.871046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.871045    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2befaa-bf7f-4806-9400-f40c87ab741c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:42.871276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.871153    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-metrics-certs\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:42.871385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.871358    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdc2\" (UniqueName: \"kubernetes.io/projected/5e2befaa-bf7f-4806-9400-f40c87ab741c-kube-api-access-hvdc2\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:42.975582 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:42.973118    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdc2\" (UniqueName: \"kubernetes.io/projected/5e2befaa-bf7f-4806-9400-f40c87ab741c-kube-api-access-hvdc2\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.001468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.001352    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.002237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.002211    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2befaa-bf7f-4806-9400-f40c87ab741c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.003300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.003273    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-metrics-certs\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.004141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.003122    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2befaa-bf7f-4806-9400-f40c87ab741c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.010465 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.010433    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-maas-api-tls\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.017756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.017240    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-metrics-certs\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.017756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.017292    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdc2\" (UniqueName: \"kubernetes.io/projected/5e2befaa-bf7f-4806-9400-f40c87ab741c-kube-api-access-hvdc2\") pod \"maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.091064 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.091028    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:43.203961 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:17:43.203900    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:17:43.384017 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.383968    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"]
Nov 03 11:17:43.385385 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:17:43.385341    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2befaa_bf7f_4806_9400_f40c87ab741c.slice/crio-52f1ec6102c90f1e2926a8d7684e0412bde447f18be558b3642a1635161445d0 WatchSource:0}: Error finding container 52f1ec6102c90f1e2926a8d7684e0412bde447f18be558b3642a1635161445d0: Status 404 returned error can't find the container with id 52f1ec6102c90f1e2926a8d7684e0412bde447f18be558b3642a1635161445d0
Nov 03 11:17:43.567056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:43.562079    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5" event={"ID":"5e2befaa-bf7f-4806-9400-f40c87ab741c","Type":"ContainerStarted","Data":"52f1ec6102c90f1e2926a8d7684e0412bde447f18be558b3642a1635161445d0"}
Nov 03 11:17:44.573092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:44.573055    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5" event={"ID":"5e2befaa-bf7f-4806-9400-f40c87ab741c","Type":"ContainerStarted","Data":"9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d"}
Nov 03 11:17:44.573607 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:44.573142    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:44.592500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:44.592444    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5" podStartSLOduration=2.035286573 podStartE2EDuration="2.592429508s" totalImagesPullingTime="557.142935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:17:44.588971119 +0000 UTC m=+2908.493825563" watchObservedRunningTime="2026-11-03 11:17:44.592429508 +0000 UTC m=+2908.497283949"
Nov 03 11:17:50.588889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:50.588860    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:17:57.730558 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:57.730478    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"]
Nov 03 11:17:57.731133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:57.730689    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" containerID="cri-o://1be0b38f6ff11d4c06abbf70677466ebdb3ded495626c50df47ba4203a1b19b3" gracePeriod=30
Nov 03 11:17:58.717560 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:58.717519    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fwgj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:17:58.717751 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:17:58.717580    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:18:02.886381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:02.886103    5393 generic.go:356] "Generic (PLEG): container finished" podID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerID="1be0b38f6ff11d4c06abbf70677466ebdb3ded495626c50df47ba4203a1b19b3" exitCode=0
Nov 03 11:18:02.886381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:02.886305    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" event={"ID":"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2","Type":"ContainerDied","Data":"1be0b38f6ff11d4c06abbf70677466ebdb3ded495626c50df47ba4203a1b19b3"}
Nov 03 11:18:03.380154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.380126    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:18:03.418903 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.418862    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"]
Nov 03 11:18:03.420193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.420170    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy"
Nov 03 11:18:03.420193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.420190    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy"
Nov 03 11:18:03.420377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.420354    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" containerName="istio-proxy"
Nov 03 11:18:03.424472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.424436    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istiod-ca-cert\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.424634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.424456    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.424634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.424566    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-kube-api-access-gmx5g\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-kube-api-access-gmx5g\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.424634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.424606    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-token\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-token\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.424960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.424935    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istiod-ca-cert" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:18:03.426281 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.426259    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-socket" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:03.429477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.429455    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9143e1-openshift-default-dockercfg-b4nvg\""
Nov 03 11:18:03.430705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.430464    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-token" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:18:03.434525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.434061    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"]
Nov 03 11:18:03.437511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.435884    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-socket\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.437511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.436007    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-certs\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.437511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.436060    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-podinfo\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.437511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.436096    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-data\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.437511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.436259    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-credential-socket\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.437511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.436310    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-envoy\") pod \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\" (UID: \"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2\") "
Nov 03 11:18:03.437964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.437528    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.437964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.437551    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.437964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.437564    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.439628 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.439600    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-certs" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:03.439628 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.439641    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-data" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:03.439958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.439827    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-credential-socket" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:03.453328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.453283    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-podinfo" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:18:03.453466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.453442    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-kube-api-access-gmx5g" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "kube-api-access-gmx5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:18:03.454050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.454028    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-envoy" pod "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" (UID: "4d98f5bc-ccdf-4f23-b64d-9612d31d89e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:03.539301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.539253    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-credential-socket\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.539507 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.539357    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-envoy\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.539606 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.539581    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-socket\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.539666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.539620    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-certs\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.539666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.539642    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-podinfo\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.539798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.539783    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmlk8\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-kube-api-access-nmlk8\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.539861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.539811    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9be7fd-4617-4579-a056-80f3bdb258cb-istiod-ca-cert\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.540005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.539977    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-data\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.540069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.540041    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-token\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.540167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.540156    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.540223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.540171    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.540223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.540181    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmx5g\" (UniqueName: \"kubernetes.io/projected/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-kube-api-access-gmx5g\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.540223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.540191    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.540223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.540201    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.540223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.540210    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:03.641688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.641652    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-token\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.641695    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-credential-socket\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.641751    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-envoy\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.641961    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-socket\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.642032    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-certs\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.642061    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-podinfo\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.642317    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-credential-socket\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.642399    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-socket\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.642604    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmlk8\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-kube-api-access-nmlk8\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.642647    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9be7fd-4617-4579-a056-80f3bdb258cb-istiod-ca-cert\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.642977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.642960    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-data\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.643335 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.643307    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-certs\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.643460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.643384    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-data\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.643654 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.643635    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9be7fd-4617-4579-a056-80f3bdb258cb-istiod-ca-cert\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.644909 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.644885    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-envoy\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.645790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.645773    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-podinfo\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.649609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.649587    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-token\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.650354 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.650333    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmlk8\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-kube-api-access-nmlk8\") pod \"e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") " pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.787604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.787250    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:03.906646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.906199    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9" event={"ID":"4d98f5bc-ccdf-4f23-b64d-9612d31d89e2","Type":"ContainerDied","Data":"39a6c78340efaf7e4abc77f6e4afa049e8e73a6c368a8fe67c9bf3e4040d6a45"}
Nov 03 11:18:03.907096 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.907069    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"
Nov 03 11:18:03.907223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.907100    5393 scope.go:122] "RemoveContainer" containerID="1be0b38f6ff11d4c06abbf70677466ebdb3ded495626c50df47ba4203a1b19b3"
Nov 03 11:18:03.940295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.940259    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"]
Nov 03 11:18:03.945083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:03.945052    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fwgj9"]
Nov 03 11:18:04.121069 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:18:04.121034    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9be7fd_4617_4579_a056_80f3bdb258cb.slice/crio-4f4d97a1fc741822fafdebf21fd607d2d6c2ec457ae62f51ba1abb00546b6615 WatchSource:0}: Error finding container 4f4d97a1fc741822fafdebf21fd607d2d6c2ec457ae62f51ba1abb00546b6615: Status 404 returned error can't find the container with id 4f4d97a1fc741822fafdebf21fd607d2d6c2ec457ae62f51ba1abb00546b6615
Nov 03 11:18:04.121609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:04.121588    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"]
Nov 03 11:18:04.124876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:04.124843    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:18:04.125008 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:04.124930    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:18:04.125008 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:04.124963    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:18:04.237295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:04.237261    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d98f5bc-ccdf-4f23-b64d-9612d31d89e2" path="/var/lib/kubelet/pods/4d98f5bc-ccdf-4f23-b64d-9612d31d89e2/volumes"
Nov 03 11:18:04.918460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:04.918425    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" event={"ID":"1d9be7fd-4617-4579-a056-80f3bdb258cb","Type":"ContainerStarted","Data":"919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910"}
Nov 03 11:18:04.918993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:04.918489    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" event={"ID":"1d9be7fd-4617-4579-a056-80f3bdb258cb","Type":"ContainerStarted","Data":"4f4d97a1fc741822fafdebf21fd607d2d6c2ec457ae62f51ba1abb00546b6615"}
Nov 03 11:18:04.942349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:04.942302    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" podStartSLOduration=10.942286364 podStartE2EDuration="10.942286364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:18:04.937480097 +0000 UTC m=+2928.842334535" watchObservedRunningTime="2026-11-03 11:18:04.942286364 +0000 UTC m=+2928.847140808"
Nov 03 11:18:05.788535 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:05.788494    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:05.796487 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:05.796451    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:05.932577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:05.932539    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:05.933758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:05.933728    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:18:08.513218 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.513184    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"]
Nov 03 11:18:08.518689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.518663    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:08.523584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.523540    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9143e1\""
Nov 03 11:18:08.523968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.523747    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9143e1-dockercfg-h8z6j\""
Nov 03 11:18:08.529356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.529329    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"]
Nov 03 11:18:08.531563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.531535    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v\" (UID: \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:08.532156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.531872    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dpwn\" (UniqueName: \"kubernetes.io/projected/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-kube-api-access-2dpwn\") pod \"payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v\" (UID: \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:08.661941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.661238    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dpwn\" (UniqueName: \"kubernetes.io/projected/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-kube-api-access-2dpwn\") pod \"payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v\" (UID: \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:08.661941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.661423    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v\" (UID: \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:08.708944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.708294    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v\" (UID: \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:08.744940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.736635    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dpwn\" (UniqueName: \"kubernetes.io/projected/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-kube-api-access-2dpwn\") pod \"payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v\" (UID: \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:08.853338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:08.853304    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:09.518968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:09.504434    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"]
Nov 03 11:18:09.518968 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:18:09.511507    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d297c0a_8957_4a5d_b652_4c11a35b9b0d.slice/crio-4fb5e3ff38201fce27a06ba1c18f8e38a48984a66b00c49c0695815ca383f665 WatchSource:0}: Error finding container 4fb5e3ff38201fce27a06ba1c18f8e38a48984a66b00c49c0695815ca383f665: Status 404 returned error can't find the container with id 4fb5e3ff38201fce27a06ba1c18f8e38a48984a66b00c49c0695815ca383f665
Nov 03 11:18:10.051100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:10.051057    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v" event={"ID":"5d297c0a-8957-4a5d-b652-4c11a35b9b0d","Type":"ContainerStarted","Data":"a0f18cf25acd65a42cb762653b42675adfe843dc02736771a0e0ef7eb6794e5c"}
Nov 03 11:18:10.051278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:10.051118    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v" event={"ID":"5d297c0a-8957-4a5d-b652-4c11a35b9b0d","Type":"ContainerStarted","Data":"4fb5e3ff38201fce27a06ba1c18f8e38a48984a66b00c49c0695815ca383f665"}
Nov 03 11:18:10.051278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:10.051149    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:10.072882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:10.072811    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v" podStartSLOduration=2.072791316 podStartE2EDuration="2.072791316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:18:10.065839051 +0000 UTC m=+2933.970693513" watchObservedRunningTime="2026-11-03 11:18:10.072791316 +0000 UTC m=+2933.977645760"
Nov 03 11:18:13.199866 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:18:13.199820    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:18:13.990992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:13.990956    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"]
Nov 03 11:18:14.046247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.046196    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.053646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.053610    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-68aa46\"/\"openshift-service-ca.crt\""
Nov 03 11:18:14.054431 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.054407    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-68aa46\"/\"test-model-68aa46-kserve-self-signed-certs\""
Nov 03 11:18:14.054597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.054432    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-68aa46\"/\"default-dockercfg-llmbn\""
Nov 03 11:18:14.055837 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.055814    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"]
Nov 03 11:18:14.059247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.059228    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-68aa46\"/\"kube-root-ca.crt\""
Nov 03 11:18:14.157440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.155993    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-dshm\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.157440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.156101    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4lm\" (UniqueName: \"kubernetes.io/projected/a6934784-f140-4ebb-af7d-b418d1238e93-kube-api-access-6p4lm\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.157440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.156240    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-model-cache\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.157440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.156409    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-tmp-dir\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.157440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.156565    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-home\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.157440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.156673    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6934784-f140-4ebb-af7d-b418d1238e93-tls-certs\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.259997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.259249    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6934784-f140-4ebb-af7d-b418d1238e93-tls-certs\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.259997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.259366    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-dshm\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.259997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.259453    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4lm\" (UniqueName: \"kubernetes.io/projected/a6934784-f140-4ebb-af7d-b418d1238e93-kube-api-access-6p4lm\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.259997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.259594    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-model-cache\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.259997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.259675    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-tmp-dir\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.259997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.259832    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-home\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.260787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.260407    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-home\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.281944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.269987    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-model-cache\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.281944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.270255    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-tmp-dir\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.281944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.279414    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-dshm\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.297940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.297576    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4lm\" (UniqueName: \"kubernetes.io/projected/a6934784-f140-4ebb-af7d-b418d1238e93-kube-api-access-6p4lm\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.319644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.319615    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6934784-f140-4ebb-af7d-b418d1238e93-tls-certs\") pod \"test-model-68aa46-kserve-6b7bcc9d47-dtmsr\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") " pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.378106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.378067    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:14.772717 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:18:14.772674    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6934784_f140_4ebb_af7d_b418d1238e93.slice/crio-39837ae9d0e4bb94afdc78385c76be75ccdb71ba92aa5756ed586a93d51b041d WatchSource:0}: Error finding container 39837ae9d0e4bb94afdc78385c76be75ccdb71ba92aa5756ed586a93d51b041d: Status 404 returned error can't find the container with id 39837ae9d0e4bb94afdc78385c76be75ccdb71ba92aa5756ed586a93d51b041d
Nov 03 11:18:14.773685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:14.773651    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"]
Nov 03 11:18:15.121281 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:15.121239    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" event={"ID":"a6934784-f140-4ebb-af7d-b418d1238e93","Type":"ContainerStarted","Data":"93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3"}
Nov 03 11:18:15.121482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:15.121306    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" event={"ID":"a6934784-f140-4ebb-af7d-b418d1238e93","Type":"ContainerStarted","Data":"39837ae9d0e4bb94afdc78385c76be75ccdb71ba92aa5756ed586a93d51b041d"}
Nov 03 11:18:15.139985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:15.139906    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" podStartSLOduration=2.139885029 podStartE2EDuration="2.139885029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:18:15.138003089 +0000 UTC m=+2939.042857545" watchObservedRunningTime="2026-11-03 11:18:15.139885029 +0000 UTC m=+2939.044739473"
Nov 03 11:18:15.380070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:15.379995    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:21.065525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:21.065432    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:18:24.164601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.164114    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"]
Nov 03 11:18:24.173950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.172455    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.177001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.176067    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"]
Nov 03 11:18:24.185169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.183866    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c0a748\"/\"default-dockercfg-xg8lq\""
Nov 03 11:18:24.185169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.183995    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c0a748\"/\"test-model-c0a748-kserve-self-signed-certs\""
Nov 03 11:18:24.185169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.184112    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c0a748\"/\"openshift-service-ca.crt\""
Nov 03 11:18:24.185169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.184205    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c0a748\"/\"kube-root-ca.crt\""
Nov 03 11:18:24.341365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.341313    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-home\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.341583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.341395    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-model-cache\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.341842 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.341814    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-tmp-dir\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.341996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.341860    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-dshm\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.341996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.341899    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nvjd\" (UniqueName: \"kubernetes.io/projected/f00dd0cb-ef27-428f-b926-21396cb38b43-kube-api-access-2nvjd\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.342477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.342449    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dd0cb-ef27-428f-b926-21396cb38b43-tls-certs\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.380296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.380263    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:24.400395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.400364    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:24.478106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.478013    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-tmp-dir\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.478370 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.478347    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-dshm\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.478528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.478512    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nvjd\" (UniqueName: \"kubernetes.io/projected/f00dd0cb-ef27-428f-b926-21396cb38b43-kube-api-access-2nvjd\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.488985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.479038    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-tmp-dir\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.498295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.498257    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dd0cb-ef27-428f-b926-21396cb38b43-tls-certs\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.498589 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.498572    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-home\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.498743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.498727    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-model-cache\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.499332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.499309    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-model-cache\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.537663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.537628    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-home\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.539934 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.539883    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nvjd\" (UniqueName: \"kubernetes.io/projected/f00dd0cb-ef27-428f-b926-21396cb38b43-kube-api-access-2nvjd\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.560614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.560580    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-dshm\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.562487 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.562460    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dd0cb-ef27-428f-b926-21396cb38b43-tls-certs\") pod \"test-model-c0a748-kserve-78d66bfc69-vz7wg\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") " pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:24.803567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:24.803469    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:25.096705 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:18:25.096671    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf00dd0cb_ef27_428f_b926_21396cb38b43.slice/crio-c69055d54b569949b51f1a2439de7ca5ffa9f65253d2ee947736f85144667993 WatchSource:0}: Error finding container c69055d54b569949b51f1a2439de7ca5ffa9f65253d2ee947736f85144667993: Status 404 returned error can't find the container with id c69055d54b569949b51f1a2439de7ca5ffa9f65253d2ee947736f85144667993
Nov 03 11:18:25.096889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:25.096869    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"]
Nov 03 11:18:25.287141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:25.287048    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg" event={"ID":"f00dd0cb-ef27-428f-b926-21396cb38b43","Type":"ContainerStarted","Data":"31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52"}
Nov 03 11:18:25.287141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:25.287099    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg" event={"ID":"f00dd0cb-ef27-428f-b926-21396cb38b43","Type":"ContainerStarted","Data":"c69055d54b569949b51f1a2439de7ca5ffa9f65253d2ee947736f85144667993"}
Nov 03 11:18:25.301176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:25.301149    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:18:25.307190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:25.307147    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg" podStartSLOduration=1.307132614 podStartE2EDuration="1.307132614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:18:25.303072314 +0000 UTC m=+2949.207926755" watchObservedRunningTime="2026-11-03 11:18:25.307132614 +0000 UTC m=+2949.211987054"
Nov 03 11:18:25.804286 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:25.804250    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:34.804550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:34.804512    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:34.818101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:34.818065    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:35.620130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:35.620097    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:18:43.193428 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:18:43.193385    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:18:44.336805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:44.336765    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"]
Nov 03 11:18:59.374150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.374106    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k" podUID="60a45c61-35ab-4b96-8758-c84a6f6d3450" containerName="main" containerID="cri-o://cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc" gracePeriod=45
Nov 03 11:18:59.686034 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.686000    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:18:59.701551 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.701503    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-dshm\") pod \"60a45c61-35ab-4b96-8758-c84a6f6d3450\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") "
Nov 03 11:18:59.701981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.701609    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a45c61-35ab-4b96-8758-c84a6f6d3450-tls-certs\" (UniqueName: \"kubernetes.io/secret/60a45c61-35ab-4b96-8758-c84a6f6d3450-tls-certs\") pod \"60a45c61-35ab-4b96-8758-c84a6f6d3450\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") "
Nov 03 11:18:59.702996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.702535    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-model-cache\") pod \"60a45c61-35ab-4b96-8758-c84a6f6d3450\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") "
Nov 03 11:18:59.702996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.702607    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-tmp-dir\") pod \"60a45c61-35ab-4b96-8758-c84a6f6d3450\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") "
Nov 03 11:18:59.702996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.702777    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-home\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-home\") pod \"60a45c61-35ab-4b96-8758-c84a6f6d3450\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") "
Nov 03 11:18:59.702996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.702886    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a45c61-35ab-4b96-8758-c84a6f6d3450-kube-api-access-8ms2x\" (UniqueName: \"kubernetes.io/projected/60a45c61-35ab-4b96-8758-c84a6f6d3450-kube-api-access-8ms2x\") pod \"60a45c61-35ab-4b96-8758-c84a6f6d3450\" (UID: \"60a45c61-35ab-4b96-8758-c84a6f6d3450\") "
Nov 03 11:18:59.707159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.704729    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-dshm" pod "60a45c61-35ab-4b96-8758-c84a6f6d3450" (UID: "60a45c61-35ab-4b96-8758-c84a6f6d3450"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:59.707159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.704987    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-model-cache" pod "60a45c61-35ab-4b96-8758-c84a6f6d3450" (UID: "60a45c61-35ab-4b96-8758-c84a6f6d3450"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:59.707159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.705896    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-tmp-dir" pod "60a45c61-35ab-4b96-8758-c84a6f6d3450" (UID: "60a45c61-35ab-4b96-8758-c84a6f6d3450"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:59.707159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.706174    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-home" pod "60a45c61-35ab-4b96-8758-c84a6f6d3450" (UID: "60a45c61-35ab-4b96-8758-c84a6f6d3450"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:18:59.710000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.709688    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a45c61-35ab-4b96-8758-c84a6f6d3450-kube-api-access-8ms2x" pod "60a45c61-35ab-4b96-8758-c84a6f6d3450" (UID: "60a45c61-35ab-4b96-8758-c84a6f6d3450"). InnerVolumeSpecName "kube-api-access-8ms2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:18:59.710138 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.710115    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a45c61-35ab-4b96-8758-c84a6f6d3450-tls-certs" pod "60a45c61-35ab-4b96-8758-c84a6f6d3450" (UID: "60a45c61-35ab-4b96-8758-c84a6f6d3450"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:18:59.805972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.805898    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ms2x\" (UniqueName: \"kubernetes.io/projected/60a45c61-35ab-4b96-8758-c84a6f6d3450-kube-api-access-8ms2x\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:59.805972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.805976    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:59.806197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.805994    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a45c61-35ab-4b96-8758-c84a6f6d3450-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:59.806197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.806007    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:59.806197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.806017    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:59.806197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.806025    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a45c61-35ab-4b96-8758-c84a6f6d3450-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:18:59.965155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.965066    5393 generic.go:356] "Generic (PLEG): container finished" podID="60a45c61-35ab-4b96-8758-c84a6f6d3450" containerID="cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc" exitCode=0
Nov 03 11:18:59.965155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.965112    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"
Nov 03 11:18:59.965155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.965143    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k" event={"ID":"60a45c61-35ab-4b96-8758-c84a6f6d3450","Type":"ContainerDied","Data":"cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc"}
Nov 03 11:18:59.965416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.965194    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k" event={"ID":"60a45c61-35ab-4b96-8758-c84a6f6d3450","Type":"ContainerDied","Data":"6088400cdf44e9b3b703c1e307dd1832f2321c51a65a71d69d1c3290e475c13a"}
Nov 03 11:18:59.965416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.965219    5393 scope.go:122] "RemoveContainer" containerID="cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc"
Nov 03 11:18:59.978828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.978807    5393 scope.go:122] "RemoveContainer" containerID="cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc"
Nov 03 11:18:59.979268 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:18:59.979236    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc\": container with ID starting with cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc not found: ID does not exist" containerID="cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc"
Nov 03 11:18:59.979370 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.979279    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc"} err="failed to get container status \"cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc\": rpc error: code = NotFound desc = could not find container \"cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc\": container with ID starting with cdcdbde96848fe9e9ad9b2835849767cee08d3859e5586ab121256b39b2eb2bc not found: ID does not exist"
Nov 03 11:18:59.990815 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.990789    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"]
Nov 03 11:18:59.995741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:18:59.995718    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9265bf/test-model-9265bf-kserve-546f64db67-lc78k"]
Nov 03 11:19:00.235616 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:00.235532    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a45c61-35ab-4b96-8758-c84a6f6d3450" path="/var/lib/kubelet/pods/60a45c61-35ab-4b96-8758-c84a6f6d3450/volumes"
Nov 03 11:19:05.453222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:05.453171    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"]
Nov 03 11:19:13.192279 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:19:13.192244    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:19:16.004531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:15.999007    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"]
Nov 03 11:19:16.131946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:16.119982    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"]
Nov 03 11:19:18.434523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:18.434373    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:19:18.439665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:18.439643    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:19:20.507659 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.507598    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p" podUID="c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" containerName="main" containerID="cri-o://d1987ed1eee83ac2e79c309abda65e0424d6e1bc0bb57e7c1b174c36001e5e33" gracePeriod=45
Nov 03 11:19:20.534114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.534071    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p" podUID="c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" containerName="main" probeResult="failure" output="Get \"https://10.134.1.1:8000/ready\": dial tcp 10.134.1.1:8000: connect: connection refused"
Nov 03 11:19:20.839739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.839701    5393 generic.go:356] "Generic (PLEG): container finished" podID="c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" containerID="d1987ed1eee83ac2e79c309abda65e0424d6e1bc0bb57e7c1b174c36001e5e33" exitCode=0
Nov 03 11:19:20.839898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.839772    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p" event={"ID":"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8","Type":"ContainerDied","Data":"d1987ed1eee83ac2e79c309abda65e0424d6e1bc0bb57e7c1b174c36001e5e33"}
Nov 03 11:19:20.839898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.839825    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p" event={"ID":"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8","Type":"ContainerDied","Data":"636d5a4923999863ec7c9c3cdf7edb58abec4dde172723be8c99aa2095277d33"}
Nov 03 11:19:20.839898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.839843    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="636d5a4923999863ec7c9c3cdf7edb58abec4dde172723be8c99aa2095277d33"
Nov 03 11:19:20.870429 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.870405    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.959081    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-home\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-home\") pod \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") "
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.959255    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-model-cache\") pod \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") "
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.959326    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tls-certs\") pod \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") "
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.959496    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-kube-api-access-lh7gm\" (UniqueName: \"kubernetes.io/projected/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-kube-api-access-lh7gm\") pod \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") "
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.959609    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-dshm\") pod \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") "
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.959656    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tmp-dir\") pod \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\" (UID: \"c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8\") "
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.960785    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tmp-dir" pod "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" (UID: "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.961033    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-home" pod "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" (UID: "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.961957    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-model-cache" pod "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" (UID: "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:20.969988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:20.965749    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tls-certs" pod "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" (UID: "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:19:21.065984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.044287    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms" podUID="09b6d65f-4cb1-4ce5-8321-a3a4b812e22d" containerName="payload-processing" containerID="cri-o://301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac" gracePeriod=25
Nov 03 11:19:21.066954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.066839    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.066954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.066873    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.066954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.066893    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.066954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.066909    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.068347 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.068273    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-dshm" pod "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" (UID: "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:21.072466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.069220    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-kube-api-access-lh7gm" pod "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" (UID: "c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8"). InnerVolumeSpecName "kube-api-access-lh7gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:21.120604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.120561    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq" podUID="237bdc8b-3f3f-4926-93bc-a23b09b58829" containerName="payload-pre-processing" containerID="cri-o://aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879" gracePeriod=25
Nov 03 11:19:21.168793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.168764    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.168793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.168790    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7gm\" (UniqueName: \"kubernetes.io/projected/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8-kube-api-access-lh7gm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.446692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.446333    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:19:21.480441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.479952    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-kube-api-access-dtk6g\" (UniqueName: \"kubernetes.io/projected/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-kube-api-access-dtk6g\") pod \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\" (UID: \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\") "
Nov 03 11:19:21.480441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.480123    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-plugins-config-volume\") pod \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\" (UID: \"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d\") "
Nov 03 11:19:21.507942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.503155    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-plugins-config-volume" pod "09b6d65f-4cb1-4ce5-8321-a3a4b812e22d" (UID: "09b6d65f-4cb1-4ce5-8321-a3a4b812e22d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:19:21.518752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.518700    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-kube-api-access-dtk6g" pod "09b6d65f-4cb1-4ce5-8321-a3a4b812e22d" (UID: "09b6d65f-4cb1-4ce5-8321-a3a4b812e22d"). InnerVolumeSpecName "kube-api-access-dtk6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:21.611949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.603142    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:19:21.611949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.605687    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtk6g\" (UniqueName: \"kubernetes.io/projected/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-kube-api-access-dtk6g\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.611949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.605724    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.707729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.707615    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237bdc8b-3f3f-4926-93bc-a23b09b58829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237bdc8b-3f3f-4926-93bc-a23b09b58829-plugins-config-volume\") pod \"237bdc8b-3f3f-4926-93bc-a23b09b58829\" (UID: \"237bdc8b-3f3f-4926-93bc-a23b09b58829\") "
Nov 03 11:19:21.707994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.707818    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237bdc8b-3f3f-4926-93bc-a23b09b58829-kube-api-access-vdlx6\" (UniqueName: \"kubernetes.io/projected/237bdc8b-3f3f-4926-93bc-a23b09b58829-kube-api-access-vdlx6\") pod \"237bdc8b-3f3f-4926-93bc-a23b09b58829\" (UID: \"237bdc8b-3f3f-4926-93bc-a23b09b58829\") "
Nov 03 11:19:21.708554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.708139    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237bdc8b-3f3f-4926-93bc-a23b09b58829-plugins-config-volume" pod "237bdc8b-3f3f-4926-93bc-a23b09b58829" (UID: "237bdc8b-3f3f-4926-93bc-a23b09b58829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:19:21.709021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.708997    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237bdc8b-3f3f-4926-93bc-a23b09b58829-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.711742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.711699    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237bdc8b-3f3f-4926-93bc-a23b09b58829-kube-api-access-vdlx6" pod "237bdc8b-3f3f-4926-93bc-a23b09b58829" (UID: "237bdc8b-3f3f-4926-93bc-a23b09b58829"). InnerVolumeSpecName "kube-api-access-vdlx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:21.821945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.821085    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlx6\" (UniqueName: \"kubernetes.io/projected/237bdc8b-3f3f-4926-93bc-a23b09b58829-kube-api-access-vdlx6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:21.880809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.880770    5393 generic.go:356] "Generic (PLEG): container finished" podID="237bdc8b-3f3f-4926-93bc-a23b09b58829" containerID="aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879" exitCode=0
Nov 03 11:19:21.881122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.880901    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq" event={"ID":"237bdc8b-3f3f-4926-93bc-a23b09b58829","Type":"ContainerDied","Data":"aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879"}
Nov 03 11:19:21.881122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.881092    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq" event={"ID":"237bdc8b-3f3f-4926-93bc-a23b09b58829","Type":"ContainerDied","Data":"1df04334c3e4c5e6dce9449ac611b25866c64dd7b1bc5124c78235ae317dc3bf"}
Nov 03 11:19:21.881269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.881134    5393 scope.go:122] "RemoveContainer" containerID="aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879"
Nov 03 11:19:21.881332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.881306    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"
Nov 03 11:19:21.920056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.917434    5393 generic.go:356] "Generic (PLEG): container finished" podID="09b6d65f-4cb1-4ce5-8321-a3a4b812e22d" containerID="301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac" exitCode=0
Nov 03 11:19:21.920056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.917712    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"
Nov 03 11:19:21.920056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.918424    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"
Nov 03 11:19:21.924980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.922375    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms" event={"ID":"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d","Type":"ContainerDied","Data":"301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac"}
Nov 03 11:19:21.924980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.922440    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms" event={"ID":"09b6d65f-4cb1-4ce5-8321-a3a4b812e22d","Type":"ContainerDied","Data":"b916630c349b138508e7cfc1aecca8ff633a0cd4fff7002d76814ea1b78429b0"}
Nov 03 11:19:21.932996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.932703    5393 scope.go:122] "RemoveContainer" containerID="aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879"
Nov 03 11:19:21.933222 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:19:21.933197    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879\": container with ID starting with aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879 not found: ID does not exist" containerID="aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879"
Nov 03 11:19:21.933370 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.933223    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879"} err="failed to get container status \"aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879\": rpc error: code = NotFound desc = could not find container \"aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879\": container with ID starting with aefd86246af8732742baa4d0a7e45400fde54bd3b14a1b7ff29ee0d088255879 not found: ID does not exist"
Nov 03 11:19:21.933370 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.933250    5393 scope.go:122] "RemoveContainer" containerID="301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac"
Nov 03 11:19:21.952775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.952755    5393 scope.go:122] "RemoveContainer" containerID="301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac"
Nov 03 11:19:21.953375 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:19:21.953345    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac\": container with ID starting with 301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac not found: ID does not exist" containerID="301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac"
Nov 03 11:19:21.953493 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.953387    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac"} err="failed to get container status \"301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac\": rpc error: code = NotFound desc = could not find container \"301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac\": container with ID starting with 301d233a57235e0645b59845ebb31bca8470fe271d78de824a0f87a67344e0ac not found: ID does not exist"
Nov 03 11:19:21.954294 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.954271    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"]
Nov 03 11:19:21.959189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.959140    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9265bf-74cf46f5ff-kx4lq"]
Nov 03 11:19:21.972415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:21.972374    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"]
Nov 03 11:19:22.009241 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:22.009189    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-60ad03/test-model-60ad03-kserve-7958b58c99-zvf8p"]
Nov 03 11:19:22.015260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:22.015230    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"]
Nov 03 11:19:22.019837 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:22.019812    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9265bf-74b4b7dbcc-7j9ms"]
Nov 03 11:19:22.234140 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:22.234063    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b6d65f-4cb1-4ce5-8321-a3a4b812e22d" path="/var/lib/kubelet/pods/09b6d65f-4cb1-4ce5-8321-a3a4b812e22d/volumes"
Nov 03 11:19:22.234542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:22.234527    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237bdc8b-3f3f-4926-93bc-a23b09b58829" path="/var/lib/kubelet/pods/237bdc8b-3f3f-4926-93bc-a23b09b58829/volumes"
Nov 03 11:19:22.235000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:22.234985    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" path="/var/lib/kubelet/pods/c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8/volumes"
Nov 03 11:19:25.857875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:25.857836    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"]
Nov 03 11:19:25.858451 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:25.858118    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy" containerID="cri-o://2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d" gracePeriod=30
Nov 03 11:19:30.270027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:30.269975    5393 generic.go:356] "Generic (PLEG): container finished" podID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerID="919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910" exitCode=0
Nov 03 11:19:30.270027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:30.270012    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" event={"ID":"1d9be7fd-4617-4579-a056-80f3bdb258cb","Type":"ContainerDied","Data":"919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910"}
Nov 03 11:19:30.270987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:30.270530    5393 scope.go:122] "RemoveContainer" containerID="919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910"
Nov 03 11:19:30.273001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:30.272967    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:19:30.273106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:30.273055    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:19:30.273106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:30.273092    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:19:30.944727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:30.944692    5393 patch_prober.go:29] interesting pod/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 11:19:30.944892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:30.944748    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 11:19:31.237366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.237335    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:19:31.379477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.373365    5393 generic.go:356] "Generic (PLEG): container finished" podID="4f889894-693a-4511-b839-d620a739a2c8" containerID="2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d" exitCode=0
Nov 03 11:19:31.379477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.373523    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" event={"ID":"4f889894-693a-4511-b839-d620a739a2c8","Type":"ContainerDied","Data":"2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d"}
Nov 03 11:19:31.379477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.373572    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs" event={"ID":"4f889894-693a-4511-b839-d620a739a2c8","Type":"ContainerDied","Data":"26c2458b6237b8b3182e9670bb7bd6afec2b697545c3e38d35b306b7e9c6724f"}
Nov 03 11:19:31.379477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.373601    5393 scope.go:122] "RemoveContainer" containerID="2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d"
Nov 03 11:19:31.379477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.373772    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"
Nov 03 11:19:31.379477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.379009    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" event={"ID":"1d9be7fd-4617-4579-a056-80f3bdb258cb","Type":"ContainerStarted","Data":"064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c"}
Nov 03 11:19:31.403186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.397683    5393 scope.go:122] "RemoveContainer" containerID="2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d"
Nov 03 11:19:31.403186 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:19:31.398139    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d\": container with ID starting with 2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d not found: ID does not exist" containerID="2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d"
Nov 03 11:19:31.403186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.398176    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d"} err="failed to get container status \"2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d\": rpc error: code = NotFound desc = could not find container \"2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d\": container with ID starting with 2ad9e5f836efa0d37edb8ef4370f194dadaacb6ba9c0a20da6cd03e83e34962d not found: ID does not exist"
Nov 03 11:19:31.459275 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459178    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-socket\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.459275 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459224    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-kube-api-access-9wnpf\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-kube-api-access-9wnpf\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.459548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459296    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-istio-token\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-istio-token\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.459548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459535    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-socket" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.459675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459576    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-data\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.459675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459601    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f889894-693a-4511-b839-d620a739a2c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f889894-693a-4511-b839-d620a739a2c8-istio-podinfo\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.459783 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459687    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-envoy\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.459783 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459727    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-certs\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.459783 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459768    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-credential-socket\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.460027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.459881    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f889894-693a-4511-b839-d620a739a2c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f889894-693a-4511-b839-d620a739a2c8-istiod-ca-cert\") pod \"4f889894-693a-4511-b839-d620a739a2c8\" (UID: \"4f889894-693a-4511-b839-d620a739a2c8\") "
Nov 03 11:19:31.460687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460337    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-data" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.460687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460285    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-certs" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.460687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460529    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-credential-socket" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.460687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460583    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f889894-693a-4511-b839-d620a739a2c8-istiod-ca-cert" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:19:31.461041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460763    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.461041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460781    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.461041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460798    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.461041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460887    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f889894-693a-4511-b839-d620a739a2c8-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.461041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.460994    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.466027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.465998    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-envoy" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:19:31.467743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.467721    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-kube-api-access-9wnpf" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "kube-api-access-9wnpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:31.468141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.468117    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f889894-693a-4511-b839-d620a739a2c8-istio-podinfo" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:19:31.469390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.469369    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-istio-token" pod "4f889894-693a-4511-b839-d620a739a2c8" (UID: "4f889894-693a-4511-b839-d620a739a2c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:31.562900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.562858    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wnpf\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-kube-api-access-9wnpf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.562900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.562900    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f889894-693a-4511-b839-d620a739a2c8-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.562900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.562931    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f889894-693a-4511-b839-d620a739a2c8-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.562900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.562947    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f889894-693a-4511-b839-d620a739a2c8-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:31.733411 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.733374    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"]
Nov 03 11:19:31.738489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.738461    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9265bf-openshift-default-795bdb684c-s7cgs"]
Nov 03 11:19:31.803031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.802988    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:19:31.807170 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.807129    5393 patch_prober.go:29] interesting pod/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Nov 03 11:19:31.807349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:31.807204    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Nov 03 11:19:32.234458 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:32.234422    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f889894-693a-4511-b839-d620a739a2c8" path="/var/lib/kubelet/pods/4f889894-693a-4511-b839-d620a739a2c8/volumes"
Nov 03 11:19:32.800944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:32.800494    5393 patch_prober.go:29] interesting pod/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Nov 03 11:19:32.800944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:32.800566    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Nov 03 11:19:33.805953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:33.805354    5393 patch_prober.go:29] interesting pod/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Nov 03 11:19:33.805953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:33.805417    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Nov 03 11:19:34.795424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:34.795380    5393 patch_prober.go:29] interesting pod/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Nov 03 11:19:34.795774 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:34.795742    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Nov 03 11:19:35.789943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:35.789316    5393 patch_prober.go:29] interesting pod/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body=
Nov 03 11:19:35.789943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:35.789397    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused"
Nov 03 11:19:35.932605 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:35.932568    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:19:36.514295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:36.514259    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"]
Nov 03 11:19:37.201332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:37.201273    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:19:37.204162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:37.204110    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:19:41.514817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:41.514751    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w" podUID="1a422450-f210-44dc-9226-43532342e514" containerName="payload-pre-processing" containerID="cri-o://14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047" gracePeriod=25
Nov 03 11:19:43.191856 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:19:43.191821    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:19:45.273102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:45.273056    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w" podUID="1a422450-f210-44dc-9226-43532342e514" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:19:55.273528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.273488    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w" podUID="1a422450-f210-44dc-9226-43532342e514" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:19:55.538262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.538159    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"]
Nov 03 11:19:55.538739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.538709    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz" podUID="ff6b8a2f-1255-4bf0-afca-616548a97736" containerName="maas-api" containerID="cri-o://a5f1774ee6c861bf00f25c74f593ce859866105872b175016b47729c32a94d55" gracePeriod=30
Nov 03 11:19:55.722832 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.722241    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"]
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739344    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739377    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739457    5393 cpu_manager.go:414] "containerMap: removing container" podUID="237bdc8b-3f3f-4926-93bc-a23b09b58829" containerName="payload-pre-processing"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739465    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237bdc8b-3f3f-4926-93bc-a23b09b58829" containerName="payload-pre-processing"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739485    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" containerName="main"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739492    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" containerName="main"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739507    5393 cpu_manager.go:414] "containerMap: removing container" podUID="09b6d65f-4cb1-4ce5-8321-a3a4b812e22d" containerName="payload-processing"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739514    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b6d65f-4cb1-4ce5-8321-a3a4b812e22d" containerName="payload-processing"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739556    5393 cpu_manager.go:414] "containerMap: removing container" podUID="60a45c61-35ab-4b96-8758-c84a6f6d3450" containerName="main"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739563    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a45c61-35ab-4b96-8758-c84a6f6d3450" containerName="main"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739790    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a45c61-35ab-4b96-8758-c84a6f6d3450" containerName="main"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739812    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f889894-693a-4511-b839-d620a739a2c8" containerName="istio-proxy"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739834    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="237bdc8b-3f3f-4926-93bc-a23b09b58829" containerName="payload-pre-processing"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739844    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71cd9f5-e96d-47c0-a7ef-35dcb86a67a8" containerName="main"
Nov 03 11:19:55.741022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.739853    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b6d65f-4cb1-4ce5-8321-a3a4b812e22d" containerName="payload-processing"
Nov 03 11:19:55.754797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.752968    5393 generic.go:356] "Generic (PLEG): container finished" podID="ff6b8a2f-1255-4bf0-afca-616548a97736" containerID="a5f1774ee6c861bf00f25c74f593ce859866105872b175016b47729c32a94d55" exitCode=0
Nov 03 11:19:55.759182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.759134    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz" event={"ID":"ff6b8a2f-1255-4bf0-afca-616548a97736","Type":"ContainerDied","Data":"a5f1774ee6c861bf00f25c74f593ce859866105872b175016b47729c32a94d55"}
Nov 03 11:19:55.759369 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.759278    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.759369 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.759196    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"]
Nov 03 11:19:55.764879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.764848    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b13404\"/\"openshift-service-ca.crt\""
Nov 03 11:19:55.765608 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.765587    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b13404\"/\"kube-root-ca.crt\""
Nov 03 11:19:55.765850 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.765831    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b13404\"/\"default-dockercfg-94284\""
Nov 03 11:19:55.766523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.766503    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b13404\"/\"test-model-b13404-kserve-self-signed-certs\""
Nov 03 11:19:55.867939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.853677    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-model-cache\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.867939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.853782    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b97e11-284c-483d-a6b3-a50073df6247-tls-certs\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.867939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.853857    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-dshm\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.867939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.859060    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrf2\" (UniqueName: \"kubernetes.io/projected/b9b97e11-284c-483d-a6b3-a50073df6247-kube-api-access-rqrf2\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.867939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.859209    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-tmp-dir\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.867939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.859495    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-home\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.963194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.963148    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-model-cache\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.963476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.963263    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b97e11-284c-483d-a6b3-a50073df6247-tls-certs\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.963476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.963355    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-dshm\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.963476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.963399    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrf2\" (UniqueName: \"kubernetes.io/projected/b9b97e11-284c-483d-a6b3-a50073df6247-kube-api-access-rqrf2\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.963739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.963532    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-tmp-dir\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.963739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.963574    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-home\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.964426 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.964193    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-home\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.964542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.964452    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-model-cache\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.967624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.967594    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-tmp-dir\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.978441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.978407    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-dshm\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.984122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.983480    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrf2\" (UniqueName: \"kubernetes.io/projected/b9b97e11-284c-483d-a6b3-a50073df6247-kube-api-access-rqrf2\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:55.991511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:55.991368    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b97e11-284c-483d-a6b3-a50073df6247-tls-certs\") pod \"test-model-b13404-kserve-77f48ffc55-5vvhm\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") " pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:56.041718 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.041687    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:19:56.064664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.064633    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-maas-api-tls\") pod \"ff6b8a2f-1255-4bf0-afca-616548a97736\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") "
Nov 03 11:19:56.064976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.064909    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6b8a2f-1255-4bf0-afca-616548a97736-kube-api-access-fv9d5\" (UniqueName: \"kubernetes.io/projected/ff6b8a2f-1255-4bf0-afca-616548a97736-kube-api-access-fv9d5\") pod \"ff6b8a2f-1255-4bf0-afca-616548a97736\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") "
Nov 03 11:19:56.065281 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.065254    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-metrics-certs\") pod \"ff6b8a2f-1255-4bf0-afca-616548a97736\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") "
Nov 03 11:19:56.065375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.065319    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6b8a2f-1255-4bf0-afca-616548a97736-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b8a2f-1255-4bf0-afca-616548a97736-openshift-service-ca\") pod \"ff6b8a2f-1255-4bf0-afca-616548a97736\" (UID: \"ff6b8a2f-1255-4bf0-afca-616548a97736\") "
Nov 03 11:19:56.066057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.065908    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6b8a2f-1255-4bf0-afca-616548a97736-openshift-service-ca" pod "ff6b8a2f-1255-4bf0-afca-616548a97736" (UID: "ff6b8a2f-1255-4bf0-afca-616548a97736"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:19:56.066584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.066559    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff6b8a2f-1255-4bf0-afca-616548a97736-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:56.068448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.068410    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-maas-api-tls" pod "ff6b8a2f-1255-4bf0-afca-616548a97736" (UID: "ff6b8a2f-1255-4bf0-afca-616548a97736"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:19:56.069508 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.069486    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b8a2f-1255-4bf0-afca-616548a97736-kube-api-access-fv9d5" pod "ff6b8a2f-1255-4bf0-afca-616548a97736" (UID: "ff6b8a2f-1255-4bf0-afca-616548a97736"). InnerVolumeSpecName "kube-api-access-fv9d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:19:56.096946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.091221    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-metrics-certs" pod "ff6b8a2f-1255-4bf0-afca-616548a97736" (UID: "ff6b8a2f-1255-4bf0-afca-616548a97736"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:19:56.169042 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.168876    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9d5\" (UniqueName: \"kubernetes.io/projected/ff6b8a2f-1255-4bf0-afca-616548a97736-kube-api-access-fv9d5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:56.169042 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.168932    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:56.169042 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.168948    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff6b8a2f-1255-4bf0-afca-616548a97736-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:19:56.187612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.187260    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:56.771991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.771952    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz" event={"ID":"ff6b8a2f-1255-4bf0-afca-616548a97736","Type":"ContainerDied","Data":"ea95d8278176bf4a4420de56f7a757037fd538e7fdcb9d97e0b5ddbc0b328e43"}
Nov 03 11:19:56.772670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.772641    5393 scope.go:122] "RemoveContainer" containerID="a5f1774ee6c861bf00f25c74f593ce859866105872b175016b47729c32a94d55"
Nov 03 11:19:56.774053 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.772003    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"
Nov 03 11:19:56.808591 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.808551    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"]
Nov 03 11:19:56.816050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.816019    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-60ad03-6f9c7dfc89-5vlxz"]
Nov 03 11:19:56.913244 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:19:56.913193    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b97e11_284c_483d_a6b3_a50073df6247.slice/crio-67f485b016103f4f9fee041ee704deabac729fdc25f551a1d84490fab266e945 WatchSource:0}: Error finding container 67f485b016103f4f9fee041ee704deabac729fdc25f551a1d84490fab266e945: Status 404 returned error can't find the container with id 67f485b016103f4f9fee041ee704deabac729fdc25f551a1d84490fab266e945
Nov 03 11:19:56.916288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:56.916248    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"]
Nov 03 11:19:57.786181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:57.786139    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm" event={"ID":"b9b97e11-284c-483d-a6b3-a50073df6247","Type":"ContainerStarted","Data":"d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a"}
Nov 03 11:19:57.786181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:57.786195    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm" event={"ID":"b9b97e11-284c-483d-a6b3-a50073df6247","Type":"ContainerStarted","Data":"67f485b016103f4f9fee041ee704deabac729fdc25f551a1d84490fab266e945"}
Nov 03 11:19:57.807540 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:57.807469    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm" podStartSLOduration=2.807452119 podStartE2EDuration="2.807452119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:19:57.806286714 +0000 UTC m=+3041.711141157" watchObservedRunningTime="2026-11-03 11:19:57.807452119 +0000 UTC m=+3041.712306561"
Nov 03 11:19:58.187789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:58.187758    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:19:58.235494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:19:58.235457    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6b8a2f-1255-4bf0-afca-616548a97736" path="/var/lib/kubelet/pods/ff6b8a2f-1255-4bf0-afca-616548a97736/volumes"
Nov 03 11:20:00.969860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:00.969821    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"]
Nov 03 11:20:00.971012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:00.970051    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" containerID="cri-o://39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492" gracePeriod=30
Nov 03 11:20:01.823712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.823683    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:20:01.868714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.868662    5393 generic.go:356] "Generic (PLEG): container finished" podID="1a422450-f210-44dc-9226-43532342e514" containerID="14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047" exitCode=0
Nov 03 11:20:01.868933 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.868828    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w" event={"ID":"1a422450-f210-44dc-9226-43532342e514","Type":"ContainerDied","Data":"14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047"}
Nov 03 11:20:01.869020 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.868951    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w" event={"ID":"1a422450-f210-44dc-9226-43532342e514","Type":"ContainerDied","Data":"985e02b1bf0b8163dcbc0b334a9dcdac1bab1ba7453a9b86088815edbd301c20"}
Nov 03 11:20:01.869020 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.868995    5393 scope.go:122] "RemoveContainer" containerID="14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047"
Nov 03 11:20:01.869238 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.869219    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"
Nov 03 11:20:01.888261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.887090    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a422450-f210-44dc-9226-43532342e514-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a422450-f210-44dc-9226-43532342e514-plugins-config-volume\") pod \"1a422450-f210-44dc-9226-43532342e514\" (UID: \"1a422450-f210-44dc-9226-43532342e514\") "
Nov 03 11:20:01.888908 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.888707    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a422450-f210-44dc-9226-43532342e514-kube-api-access-t4hpv\" (UniqueName: \"kubernetes.io/projected/1a422450-f210-44dc-9226-43532342e514-kube-api-access-t4hpv\") pod \"1a422450-f210-44dc-9226-43532342e514\" (UID: \"1a422450-f210-44dc-9226-43532342e514\") "
Nov 03 11:20:01.889438 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.889398    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a422450-f210-44dc-9226-43532342e514-plugins-config-volume" pod "1a422450-f210-44dc-9226-43532342e514" (UID: "1a422450-f210-44dc-9226-43532342e514"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:20:01.892720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.892687    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a422450-f210-44dc-9226-43532342e514-kube-api-access-t4hpv" pod "1a422450-f210-44dc-9226-43532342e514" (UID: "1a422450-f210-44dc-9226-43532342e514"). InnerVolumeSpecName "kube-api-access-t4hpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:01.894150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.894126    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a422450-f210-44dc-9226-43532342e514-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:01.894311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.894296    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4hpv\" (UniqueName: \"kubernetes.io/projected/1a422450-f210-44dc-9226-43532342e514-kube-api-access-t4hpv\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:01.922183 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.922137    5393 scope.go:122] "RemoveContainer" containerID="14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047"
Nov 03 11:20:01.922609 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:20:01.922579    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047\": container with ID starting with 14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047 not found: ID does not exist" containerID="14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047"
Nov 03 11:20:01.922764 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:01.922734    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047"} err="failed to get container status \"14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047\": rpc error: code = NotFound desc = could not find container \"14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047\": container with ID starting with 14b15eb6f5d649052d306830753b7df6a488075c2bfd241249c2cbe0f89e7047 not found: ID does not exist"
Nov 03 11:20:02.253546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:02.253503    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"]
Nov 03 11:20:02.259496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:02.259459    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-60ad03-666dbb97d7-g8x4w"]
Nov 03 11:20:03.278996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:03.278966    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"]
Nov 03 11:20:03.279521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:03.279181    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy" containerID="cri-o://4e9de5faae999a414abcbf52041f6a100abc45b72911376062ab338864c4b286" gracePeriod=30
Nov 03 11:20:04.234143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:04.234107    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a422450-f210-44dc-9226-43532342e514" path="/var/lib/kubelet/pods/1a422450-f210-44dc-9226-43532342e514/volumes"
Nov 03 11:20:06.194147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.194109    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:20:06.210015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.209977    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:20:06.358672 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.358636    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 11:20:06.369123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369093    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-podinfo\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.369308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369154    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-kube-api-access-vv7jl\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.369308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369232    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-token\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-token\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.369308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369259    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-envoy\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.369308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369281    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-credential-socket\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.369531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369355    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-data\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.369531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369432    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istiod-ca-cert\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.369640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369545    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-socket\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.369772 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.369749    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-certs\") pod \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\" (UID: \"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690\") "
Nov 03 11:20:06.370313 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.370285    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-data" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:06.370488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.370434    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-socket" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:06.370649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.370492    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istiod-ca-cert" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:20:06.370649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.370553    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-credential-socket" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:06.370836 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.370816    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-certs" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:06.371382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.371363    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.371495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.371389    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.371495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.371402    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.371495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.371416    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.371495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.371428    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.372143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.372117    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-envoy" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:06.372332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.372311    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-podinfo" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:20:06.372485 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.372467    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-token" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:06.372669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.372656    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-kube-api-access-vv7jl" pod "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" (UID: "6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690"). InnerVolumeSpecName "kube-api-access-vv7jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:06.473590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.473555    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.473590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.473588    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-kube-api-access-vv7jl\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.473590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.473598    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.473911 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.473607    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:06.980587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.980554    5393 generic.go:356] "Generic (PLEG): container finished" podID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerID="39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492" exitCode=0
Nov 03 11:20:06.980794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.980597    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" event={"ID":"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690","Type":"ContainerDied","Data":"39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492"}
Nov 03 11:20:06.980794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.980628    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"
Nov 03 11:20:06.980794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.980669    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" event={"ID":"6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690","Type":"ContainerDied","Data":"dc881e4a757b969308405023cebb51e8e2df99145373737c210eebf23ef95628"}
Nov 03 11:20:06.980794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.980700    5393 scope.go:122] "RemoveContainer" containerID="39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492"
Nov 03 11:20:06.996413 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:06.996387    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:20:07.002475 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:07.002455    5393 scope.go:122] "RemoveContainer" containerID="39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492"
Nov 03 11:20:07.002756 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:20:07.002736    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492\": container with ID starting with 39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492 not found: ID does not exist" containerID="39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492"
Nov 03 11:20:07.002803 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:07.002766    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492"} err="failed to get container status \"39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492\": rpc error: code = NotFound desc = could not find container \"39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492\": container with ID starting with 39309f95f77f4198279072fb3af7296d15c9db4c06b05104e05c19d6c8ff0492 not found: ID does not exist"
Nov 03 11:20:07.009470 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:07.009449    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"]
Nov 03 11:20:07.014295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:07.014262    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr"]
Nov 03 11:20:07.354223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:07.354185    5393 patch_prober.go:29] interesting pod/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:20:07.354681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:07.354257    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010232-openshift-default-5bfcc95779-8bgnr" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:20:08.112016 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.111985    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"]
Nov 03 11:20:08.113604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.113570    5393 cpu_manager.go:414] "containerMap: removing container" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy"
Nov 03 11:20:08.113604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.113602    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy"
Nov 03 11:20:08.113821 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.113646    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1a422450-f210-44dc-9226-43532342e514" containerName="payload-pre-processing"
Nov 03 11:20:08.113821 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.113654    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a422450-f210-44dc-9226-43532342e514" containerName="payload-pre-processing"
Nov 03 11:20:08.113821 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.113681    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b8a2f-1255-4bf0-afca-616548a97736" containerName="maas-api"
Nov 03 11:20:08.113821 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.113688    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b8a2f-1255-4bf0-afca-616548a97736" containerName="maas-api"
Nov 03 11:20:08.114083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.114003    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" containerName="istio-proxy"
Nov 03 11:20:08.114083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.114031    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a422450-f210-44dc-9226-43532342e514" containerName="payload-pre-processing"
Nov 03 11:20:08.114083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.114045    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b8a2f-1255-4bf0-afca-616548a97736" containerName="maas-api"
Nov 03 11:20:08.129093 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.129062    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"]
Nov 03 11:20:08.129380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.129353    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.137226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.136611    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9143e1\"/\"test-model-9143e1-kserve-self-signed-certs\""
Nov 03 11:20:08.137226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.136808    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9143e1\"/\"kube-root-ca.crt\""
Nov 03 11:20:08.137226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.136884    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9143e1\"/\"default-dockercfg-99k4p\""
Nov 03 11:20:08.137226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.136999    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9143e1\"/\"openshift-service-ca.crt\""
Nov 03 11:20:08.210705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.210673    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-dshm\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.210992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.210729    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-tmp-dir\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.210992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.210957    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-home\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.211160 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.211107    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvk7t\" (UniqueName: \"kubernetes.io/projected/ea7380ea-3664-469f-a0d4-c947bdecb710-kube-api-access-xvk7t\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.211310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.211283    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7380ea-3664-469f-a0d4-c947bdecb710-tls-certs\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.211465 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.211440    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-model-cache\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.242942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.239730    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690" path="/var/lib/kubelet/pods/6ee3e75b-fb64-4bf0-86cc-df7dfc4a2690/volumes"
Nov 03 11:20:08.327955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.326440    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-dshm\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.327955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.326531    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-tmp-dir\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.327955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.326631    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-home\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.327955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.326728    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvk7t\" (UniqueName: \"kubernetes.io/projected/ea7380ea-3664-469f-a0d4-c947bdecb710-kube-api-access-xvk7t\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.327955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.326861    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7380ea-3664-469f-a0d4-c947bdecb710-tls-certs\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.327955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.327009    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-model-cache\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.327955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.327552    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-model-cache\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.345941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.338287    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-home\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.345941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.338578    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-tmp-dir\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.358953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.355985    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvk7t\" (UniqueName: \"kubernetes.io/projected/ea7380ea-3664-469f-a0d4-c947bdecb710-kube-api-access-xvk7t\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.382428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.382255    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-dshm\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.382880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.382849    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7380ea-3664-469f-a0d4-c947bdecb710-tls-certs\") pod \"test-model-9143e1-kserve-7bb7d77bc7-hcl27\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") " pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.447532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.447497    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:08.870318 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:20:08.870284    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea7380ea_3664_469f_a0d4_c947bdecb710.slice/crio-e04712e14f1bebc9b9d07879dcc8eca8b6209701044c6e6e76425a92b5dd2b4e WatchSource:0}: Error finding container e04712e14f1bebc9b9d07879dcc8eca8b6209701044c6e6e76425a92b5dd2b4e: Status 404 returned error can't find the container with id e04712e14f1bebc9b9d07879dcc8eca8b6209701044c6e6e76425a92b5dd2b4e
Nov 03 11:20:08.873106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:08.873079    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"]
Nov 03 11:20:09.080592 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.078484    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27" event={"ID":"ea7380ea-3664-469f-a0d4-c947bdecb710","Type":"ContainerStarted","Data":"485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072"}
Nov 03 11:20:09.080592 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.078541    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27" event={"ID":"ea7380ea-3664-469f-a0d4-c947bdecb710","Type":"ContainerStarted","Data":"e04712e14f1bebc9b9d07879dcc8eca8b6209701044c6e6e76425a92b5dd2b4e"}
Nov 03 11:20:09.083454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.083421    5393 generic.go:356] "Generic (PLEG): container finished" podID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerID="4e9de5faae999a414abcbf52041f6a100abc45b72911376062ab338864c4b286" exitCode=0
Nov 03 11:20:09.083929 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.083891    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" event={"ID":"cdf58613-07c6-4c00-98c2-697d8763ed79","Type":"ContainerDied","Data":"4e9de5faae999a414abcbf52041f6a100abc45b72911376062ab338864c4b286"}
Nov 03 11:20:09.112463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.112410    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27" podStartSLOduration=1.112396112 podStartE2EDuration="1.112396112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:20:09.111362103 +0000 UTC m=+3053.016216546" watchObservedRunningTime="2026-11-03 11:20:09.112396112 +0000 UTC m=+3053.017250552"
Nov 03 11:20:09.164478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.164449    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 11:20:09.182639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.182606    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-podinfo\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.182820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.182668    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-certs\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.182820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.182774    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf58613-07c6-4c00-98c2-697d8763ed79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf58613-07c6-4c00-98c2-697d8763ed79-istiod-ca-cert\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.182968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.182948    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-certs" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:09.183197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.183167    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf58613-07c6-4c00-98c2-697d8763ed79-istiod-ca-cert" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:20:09.183372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.182908    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-token\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-token\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.183648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.183474    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-data\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.183648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.183524    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-credential-socket\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.183648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.183596    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-kube-api-access-9x2b6\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-kube-api-access-9x2b6\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.183979 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.183680    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-envoy\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.183979 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.183733    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-socket\") pod \"cdf58613-07c6-4c00-98c2-697d8763ed79\" (UID: \"cdf58613-07c6-4c00-98c2-697d8763ed79\") "
Nov 03 11:20:09.184160 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.183969    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-credential-socket" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:09.184822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.184800    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.192098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.192074    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.192230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.192100    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf58613-07c6-4c00-98c2-697d8763ed79-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.192230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.185416    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-socket" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:09.192230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.186606    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-data" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:09.192230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.188586    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-kube-api-access-9x2b6" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "kube-api-access-9x2b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:09.192230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.189023    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-token" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:09.192230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.189074    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-envoy" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:09.192603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.192363    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-podinfo" pod "cdf58613-07c6-4c00-98c2-697d8763ed79" (UID: "cdf58613-07c6-4c00-98c2-697d8763ed79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:20:09.294276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.294244    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.294276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.294272    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.294276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.294282    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.294601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.294291    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x2b6\" (UniqueName: \"kubernetes.io/projected/cdf58613-07c6-4c00-98c2-697d8763ed79-kube-api-access-9x2b6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.294601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.294300    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.294601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.294308    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf58613-07c6-4c00-98c2-697d8763ed79-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:09.448137 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:09.448054    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:10.094712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:10.094684    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"
Nov 03 11:20:10.094955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:10.094719    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn" event={"ID":"cdf58613-07c6-4c00-98c2-697d8763ed79","Type":"ContainerDied","Data":"dec56472d66a8fa0f0bfe22d44fbebb41c30efe037fd19dba71f7d9e4dfd1837"}
Nov 03 11:20:10.094955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:10.094794    5393 scope.go:122] "RemoveContainer" containerID="4e9de5faae999a414abcbf52041f6a100abc45b72911376062ab338864c4b286"
Nov 03 11:20:10.122300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:10.122255    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"]
Nov 03 11:20:10.126679 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:10.126655    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-5hnmn"]
Nov 03 11:20:10.237399 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:10.237365    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" path="/var/lib/kubelet/pods/cdf58613-07c6-4c00-98c2-697d8763ed79/volumes"
Nov 03 11:20:12.740152 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:12.740115    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"]
Nov 03 11:20:12.740642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:12.740418    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" containerID="cri-o://cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc" gracePeriod=30
Nov 03 11:20:13.193353 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:20:13.193317    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:20:14.441395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:14.441355    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fcrct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:20:14.441860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:14.441421    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:20:18.100025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.096831    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:20:18.194045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.194001    5393 generic.go:356] "Generic (PLEG): container finished" podID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerID="cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc" exitCode=0
Nov 03 11:20:18.194223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.194055    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"
Nov 03 11:20:18.194223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.194082    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" event={"ID":"4ef6c43b-90bb-4158-8036-1c3b70f16396","Type":"ContainerDied","Data":"cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc"}
Nov 03 11:20:18.194223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.194140    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct" event={"ID":"4ef6c43b-90bb-4158-8036-1c3b70f16396","Type":"ContainerDied","Data":"0a0cc2fdfaf3c305a88f49fe39ca1dab8efc66decbeedb08210f07efa4438ba9"}
Nov 03 11:20:18.194223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.194166    5393 scope.go:122] "RemoveContainer" containerID="cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc"
Nov 03 11:20:18.201019 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.200983    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-credential-socket\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201158    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-envoy\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201194    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-token\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-token\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201278    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-podinfo\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201307    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef6c43b-90bb-4158-8036-1c3b70f16396-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef6c43b-90bb-4158-8036-1c3b70f16396-istiod-ca-cert\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201440    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-socket\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201469    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-certs\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201588    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-kube-api-access-858zx\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-kube-api-access-858zx\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201670    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-data\") pod \"4ef6c43b-90bb-4158-8036-1c3b70f16396\" (UID: \"4ef6c43b-90bb-4158-8036-1c3b70f16396\") "
Nov 03 11:20:18.201884 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201778    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-socket" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:18.201884 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.201778    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef6c43b-90bb-4158-8036-1c3b70f16396-istiod-ca-cert" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:20:18.202157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.202124    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-certs" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:18.202240 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.202161    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-data" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:18.202555 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.202531    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-credential-socket" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:18.202721 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.202700    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.202798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.202728    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.202798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.202744    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.202798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.202758    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.202798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.202771    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef6c43b-90bb-4158-8036-1c3b70f16396-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.204641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.204610    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-envoy" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:20:18.205222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.205199    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-kube-api-access-858zx" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "kube-api-access-858zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:18.207130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.207093    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-podinfo" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:20:18.213241 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.213205    5393 scope.go:122] "RemoveContainer" containerID="cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc"
Nov 03 11:20:18.213712 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:20:18.213685    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc\": container with ID starting with cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc not found: ID does not exist" containerID="cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc"
Nov 03 11:20:18.213829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.213722    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc"} err="failed to get container status \"cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc\": rpc error: code = NotFound desc = could not find container \"cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc\": container with ID starting with cc4745aa4c43477592feaa2c2ac2c83f2d06c1ce2618f6881f2ae74dc49a24cc not found: ID does not exist"
Nov 03 11:20:18.214904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.214881    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-token" pod "4ef6c43b-90bb-4158-8036-1c3b70f16396" (UID: "4ef6c43b-90bb-4158-8036-1c3b70f16396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:20:18.304224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.304145    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.304224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.304174    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-858zx\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-kube-api-access-858zx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.304224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.304184    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.304224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.304194    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef6c43b-90bb-4158-8036-1c3b70f16396-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:20:18.448794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.448758    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:18.463295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.463267    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:18.515256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.515224    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"]
Nov 03 11:20:18.520034 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:18.520007    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fcrct"]
Nov 03 11:20:19.216236 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:19.216205    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:20:20.233604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:20.233563    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" path="/var/lib/kubelet/pods/4ef6c43b-90bb-4158-8036-1c3b70f16396/volumes"
Nov 03 11:20:20.768130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:20.768097    5393 scope.go:122] "RemoveContainer" containerID="d1987ed1eee83ac2e79c309abda65e0424d6e1bc0bb57e7c1b174c36001e5e33"
Nov 03 11:20:43.196062 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:20:43.196021    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:20:53.895042 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.894411    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"]
Nov 03 11:20:53.895867 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.895838    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy"
Nov 03 11:20:53.895867 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.895865    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy"
Nov 03 11:20:53.896169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.895907    5393 cpu_manager.go:414] "containerMap: removing container" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy"
Nov 03 11:20:53.896169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.895932    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy"
Nov 03 11:20:53.896310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.896225    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef6c43b-90bb-4158-8036-1c3b70f16396" containerName="istio-proxy"
Nov 03 11:20:53.896310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.896250    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf58613-07c6-4c00-98c2-697d8763ed79" containerName="istio-proxy"
Nov 03 11:20:53.914984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.904215    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.914984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.909954    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"]
Nov 03 11:20:53.914984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.914296    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b13404-openshift-default-dockercfg-bxcqw\""
Nov 03 11:20:53.963530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.963500    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.963741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.963655    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.963827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.963757    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.963892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.963866    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.963981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.963907    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.964052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.963999    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a685fa6-5b4a-45bc-8877-815f154949d3-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.964113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.964059    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.964173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.964125    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:53.964265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:53.964246    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqc8p\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-kube-api-access-fqc8p\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.065619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.065575    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.065827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.065673    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.065827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.065729    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.065907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.065823    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.065907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.065860    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.065907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.065896    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a685fa6-5b4a-45bc-8877-815f154949d3-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.066134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.065975    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.066134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.066018    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.066134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.066080    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-data\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.066305 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.066193    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.066305 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.066194    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqc8p\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-kube-api-access-fqc8p\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.066432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.066312    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-credential-socket\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.066432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.066366    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-certs\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.066715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.066690    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a685fa6-5b4a-45bc-8877-815f154949d3-istiod-ca-cert\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.068586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.068561    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-envoy\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.068857 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.068836    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-podinfo\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.074476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.074449    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-token\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.074570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.074557    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqc8p\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-kube-api-access-fqc8p\") pod \"e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") " pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.233529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.233439    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:54.536591 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:20:54.536560    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a685fa6_5b4a_45bc_8877_815f154949d3.slice/crio-2d87bccea3063bac53b06a86f3a7606ec8236c1531d5d5834e1ee223022598a7 WatchSource:0}: Error finding container 2d87bccea3063bac53b06a86f3a7606ec8236c1531d5d5834e1ee223022598a7: Status 404 returned error can't find the container with id 2d87bccea3063bac53b06a86f3a7606ec8236c1531d5d5834e1ee223022598a7
Nov 03 11:20:54.536885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.536861    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"]
Nov 03 11:20:54.539762 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.539728    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:20:54.539870 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.539806    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:20:54.539870 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.539851    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:20:54.645828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.645786    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" event={"ID":"6a685fa6-5b4a-45bc-8877-815f154949d3","Type":"ContainerStarted","Data":"55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16"}
Nov 03 11:20:54.646007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.645851    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" event={"ID":"6a685fa6-5b4a-45bc-8877-815f154949d3","Type":"ContainerStarted","Data":"2d87bccea3063bac53b06a86f3a7606ec8236c1531d5d5834e1ee223022598a7"}
Nov 03 11:20:54.732162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:54.732106    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podStartSLOduration=1.7320893389999998 podStartE2EDuration="1.732089339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:20:54.677798249 +0000 UTC m=+3098.582652693" watchObservedRunningTime="2026-11-03 11:20:54.732089339 +0000 UTC m=+3098.636943779"
Nov 03 11:20:55.234634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:55.234593    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:20:55.236909 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:55.236882    5393 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 03 11:20:55.237047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:55.236965    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 03 11:20:56.234722 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:56.234691    5393 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 03 11:20:56.235200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:56.234736    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 03 11:20:57.234440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:57.234402    5393 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 03 11:20:57.234633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:57.234468    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 03 11:20:58.233859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:58.233828    5393 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 03 11:20:58.234281 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:58.233879    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 03 11:20:59.233848 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:59.233809    5393 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 03 11:20:59.234053 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:20:59.233878    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 03 11:21:00.234052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:00.234021    5393 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 03 11:21:00.234487 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:00.234082    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 03 11:21:01.234526 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:01.234493    5393 patch_prober.go:29] interesting pod/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 03 11:21:01.235076 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:01.234572    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 03 11:21:02.240053 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:02.240019    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:21:02.240565 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:02.240310    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:21:02.241254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:02.241235    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:21:12.819271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.819228    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"]
Nov 03 11:21:12.824352 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.824332    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.836315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.836286    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"]
Nov 03 11:21:12.939822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.939777    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.939822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.939830    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.940101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.940005    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.940202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.940185    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.940254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.940238    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.940512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.940491    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6h7\" (UniqueName: \"kubernetes.io/projected/56fd579f-dff9-4b92-ac15-00bdf72f86a8-kube-api-access-ll6h7\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.940557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.940531    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.940595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.940556    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:12.940673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:12.940651    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.042630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.042586    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.042845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.042726    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.042845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.042788    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043029    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6h7\" (UniqueName: \"kubernetes.io/projected/56fd579f-dff9-4b92-ac15-00bdf72f86a8-kube-api-access-ll6h7\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043065    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043090    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043177    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043284 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043257    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043347 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043261    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043347 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043328    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043457 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043378    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043457 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043407    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043659    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.043729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.043696    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.045550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.045528    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.045666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.045648    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.051078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.051055    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6h7\" (UniqueName: \"kubernetes.io/projected/56fd579f-dff9-4b92-ac15-00bdf72f86a8-kube-api-access-ll6h7\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.051235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.051215    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fd579f-dff9-4b92-ac15-00bdf72f86a8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fv94c\" (UID: \"56fd579f-dff9-4b92-ac15-00bdf72f86a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.156671 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.156394    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:13.230742 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:21:13.230629    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:21:13.478682 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.478632    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"]
Nov 03 11:21:13.479294 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:21:13.479267    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fd579f_dff9_4b92_ac15_00bdf72f86a8.slice/crio-e23db06e550479c5479c23cee4ab1328890440a52aba2c77067993e425e6695f WatchSource:0}: Error finding container e23db06e550479c5479c23cee4ab1328890440a52aba2c77067993e425e6695f: Status 404 returned error can't find the container with id e23db06e550479c5479c23cee4ab1328890440a52aba2c77067993e425e6695f
Nov 03 11:21:13.482989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.482891    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:21:13.483110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.483089    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:21:13.483157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.483142    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:21:13.940939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.940432    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" event={"ID":"56fd579f-dff9-4b92-ac15-00bdf72f86a8","Type":"ContainerStarted","Data":"5d304705c87cfb463b1d546b2f582a054f7793771ce27504dc02d0d18b21eca1"}
Nov 03 11:21:13.940939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.940500    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" event={"ID":"56fd579f-dff9-4b92-ac15-00bdf72f86a8","Type":"ContainerStarted","Data":"e23db06e550479c5479c23cee4ab1328890440a52aba2c77067993e425e6695f"}
Nov 03 11:21:13.966499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:13.966425    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podStartSLOduration=1.9664084179999999 podStartE2EDuration="1.966408418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:21:13.962902494 +0000 UTC m=+3117.867756935" watchObservedRunningTime="2026-11-03 11:21:13.966408418 +0000 UTC m=+3117.871262858"
Nov 03 11:21:14.158030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:14.157995    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:14.159578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:14.159554    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fv94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Nov 03 11:21:14.159658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:14.159603    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podUID="56fd579f-dff9-4b92-ac15-00bdf72f86a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Nov 03 11:21:15.158161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:15.158118    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fv94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Nov 03 11:21:15.158600 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:15.158179    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podUID="56fd579f-dff9-4b92-ac15-00bdf72f86a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Nov 03 11:21:15.784947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:15.782792    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"]
Nov 03 11:21:16.157622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:16.157587    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fv94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Nov 03 11:21:16.157966 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:16.157908    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podUID="56fd579f-dff9-4b92-ac15-00bdf72f86a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Nov 03 11:21:17.158158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.158121    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fv94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Nov 03 11:21:17.158665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.158192    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podUID="56fd579f-dff9-4b92-ac15-00bdf72f86a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Nov 03 11:21:17.345301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.345262    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"]
Nov 03 11:21:17.350372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.350346    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.356804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.356766    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:21:17.357761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.357717    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"]
Nov 03 11:21:17.358235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.358142    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:21:17.470064 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.469963    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92zd7\" (UniqueName: \"kubernetes.io/projected/5630cb64-48b7-4aa0-8cf1-64f867be7f06-kube-api-access-92zd7\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.470233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.470121    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.470502 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.470450    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.656957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.656906    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.657153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.657096    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92zd7\" (UniqueName: \"kubernetes.io/projected/5630cb64-48b7-4aa0-8cf1-64f867be7f06-kube-api-access-92zd7\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.657153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.657132    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.657812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.657790    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.659524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.659501    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.664769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.664748    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92zd7\" (UniqueName: \"kubernetes.io/projected/5630cb64-48b7-4aa0-8cf1-64f867be7f06-kube-api-access-92zd7\") pod \"maas-api-revoke-sub-637e41cb-x697v\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:17.674654 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:17.674633    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:18.024214 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:18.018484    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"]
Nov 03 11:21:18.158435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:18.158402    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fv94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Nov 03 11:21:18.158865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:18.158454    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podUID="56fd579f-dff9-4b92-ac15-00bdf72f86a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Nov 03 11:21:19.075234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:19.075189    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v" event={"ID":"5630cb64-48b7-4aa0-8cf1-64f867be7f06","Type":"ContainerStarted","Data":"4c4fcee5b8b76b9139142b0ce04893b970aa6d8747ff41e0099044c659910ba5"}
Nov 03 11:21:19.075457 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:19.075253    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v" event={"ID":"5630cb64-48b7-4aa0-8cf1-64f867be7f06","Type":"ContainerStarted","Data":"2f5efed2ac1034b824e54f5a57c8daecd15d4abc72e8d9cb5b88113ddc9c6269"}
Nov 03 11:21:19.113953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:19.112450    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v" podStartSLOduration=2.112426147 podStartE2EDuration="2.112426147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:21:19.095329791 +0000 UTC m=+3123.000184233" watchObservedRunningTime="2026-11-03 11:21:19.112426147 +0000 UTC m=+3123.017280589"
Nov 03 11:21:19.158023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:19.157988    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fv94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Nov 03 11:21:19.158222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:19.158049    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podUID="56fd579f-dff9-4b92-ac15-00bdf72f86a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Nov 03 11:21:20.100797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:20.100759    5393 generic.go:356] "Generic (PLEG): container finished" podID="5630cb64-48b7-4aa0-8cf1-64f867be7f06" containerID="4c4fcee5b8b76b9139142b0ce04893b970aa6d8747ff41e0099044c659910ba5" exitCode=0
Nov 03 11:21:20.101299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:20.100847    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v" event={"ID":"5630cb64-48b7-4aa0-8cf1-64f867be7f06","Type":"ContainerDied","Data":"4c4fcee5b8b76b9139142b0ce04893b970aa6d8747ff41e0099044c659910ba5"}
Nov 03 11:21:20.158238 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:20.158204    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fv94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Nov 03 11:21:20.158419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:20.158278    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podUID="56fd579f-dff9-4b92-ac15-00bdf72f86a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Nov 03 11:21:21.157556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.157516    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fv94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused" start-of-body=
Nov 03 11:21:21.158062 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.157602    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c" podUID="56fd579f-dff9-4b92-ac15-00bdf72f86a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.42:15021/healthz/ready\": dial tcp 10.134.1.42:15021: connect: connection refused"
Nov 03 11:21:21.402006 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.401889    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:21.490296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.490261    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-openshift-ca\") pod \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") "
Nov 03 11:21:21.490542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.490385    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-xks-ca\") pod \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") "
Nov 03 11:21:21.490735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.490708    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5630cb64-48b7-4aa0-8cf1-64f867be7f06-kube-api-access-92zd7\" (UniqueName: \"kubernetes.io/projected/5630cb64-48b7-4aa0-8cf1-64f867be7f06-kube-api-access-92zd7\") pod \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\" (UID: \"5630cb64-48b7-4aa0-8cf1-64f867be7f06\") "
Nov 03 11:21:21.490810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.490783    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-openshift-ca" pod "5630cb64-48b7-4aa0-8cf1-64f867be7f06" (UID: "5630cb64-48b7-4aa0-8cf1-64f867be7f06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:21.492023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.491973    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:21.493660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.493626    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-xks-ca" pod "5630cb64-48b7-4aa0-8cf1-64f867be7f06" (UID: "5630cb64-48b7-4aa0-8cf1-64f867be7f06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:21.494354 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.494324    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5630cb64-48b7-4aa0-8cf1-64f867be7f06-kube-api-access-92zd7" pod "5630cb64-48b7-4aa0-8cf1-64f867be7f06" (UID: "5630cb64-48b7-4aa0-8cf1-64f867be7f06"). InnerVolumeSpecName "kube-api-access-92zd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:21.594374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.594329    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92zd7\" (UniqueName: \"kubernetes.io/projected/5630cb64-48b7-4aa0-8cf1-64f867be7f06-kube-api-access-92zd7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:21.594374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:21.594362    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5630cb64-48b7-4aa0-8cf1-64f867be7f06-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:22.122821 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:22.122780    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v" event={"ID":"5630cb64-48b7-4aa0-8cf1-64f867be7f06","Type":"ContainerDied","Data":"2f5efed2ac1034b824e54f5a57c8daecd15d4abc72e8d9cb5b88113ddc9c6269"}
Nov 03 11:21:22.123038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:22.122841    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"
Nov 03 11:21:22.123038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:22.122850    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f5efed2ac1034b824e54f5a57c8daecd15d4abc72e8d9cb5b88113ddc9c6269"
Nov 03 11:21:22.773433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:22.773397    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:22.774299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:22.774278    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:22.775360 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:22.775341    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fv94c"
Nov 03 11:21:26.967943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:26.962137    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"]
Nov 03 11:21:31.038665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.038554    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq" podUID="c7a21254-4030-4431-8334-4ac98d53c6ab" containerName="main" containerID="cri-o://13912600d24687c351e91b0dc40cf371fd25f2188dc6f82e11a29197c286b1c1" gracePeriod=45
Nov 03 11:21:31.324511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.324295    5393 generic.go:356] "Generic (PLEG): container finished" podID="c7a21254-4030-4431-8334-4ac98d53c6ab" containerID="13912600d24687c351e91b0dc40cf371fd25f2188dc6f82e11a29197c286b1c1" exitCode=0
Nov 03 11:21:31.324511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.324421    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq" event={"ID":"c7a21254-4030-4431-8334-4ac98d53c6ab","Type":"ContainerDied","Data":"13912600d24687c351e91b0dc40cf371fd25f2188dc6f82e11a29197c286b1c1"}
Nov 03 11:21:31.445123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.445095    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:21:31.589071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.588974    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a21254-4030-4431-8334-4ac98d53c6ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a21254-4030-4431-8334-4ac98d53c6ab-tls-certs\") pod \"c7a21254-4030-4431-8334-4ac98d53c6ab\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") "
Nov 03 11:21:31.589282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.589128    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-home\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-home\") pod \"c7a21254-4030-4431-8334-4ac98d53c6ab\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") "
Nov 03 11:21:31.589774 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.589401    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a21254-4030-4431-8334-4ac98d53c6ab-kube-api-access-pmx82\" (UniqueName: \"kubernetes.io/projected/c7a21254-4030-4431-8334-4ac98d53c6ab-kube-api-access-pmx82\") pod \"c7a21254-4030-4431-8334-4ac98d53c6ab\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") "
Nov 03 11:21:31.589935 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.589863    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-dshm\") pod \"c7a21254-4030-4431-8334-4ac98d53c6ab\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") "
Nov 03 11:21:31.590038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.589989    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-model-cache\") pod \"c7a21254-4030-4431-8334-4ac98d53c6ab\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") "
Nov 03 11:21:31.590131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.590080    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-tmp-dir\") pod \"c7a21254-4030-4431-8334-4ac98d53c6ab\" (UID: \"c7a21254-4030-4431-8334-4ac98d53c6ab\") "
Nov 03 11:21:31.590642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.590561    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-tmp-dir" pod "c7a21254-4030-4431-8334-4ac98d53c6ab" (UID: "c7a21254-4030-4431-8334-4ac98d53c6ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:31.590642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.590607    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-model-cache" pod "c7a21254-4030-4431-8334-4ac98d53c6ab" (UID: "c7a21254-4030-4431-8334-4ac98d53c6ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:31.591434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.591407    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-home" pod "c7a21254-4030-4431-8334-4ac98d53c6ab" (UID: "c7a21254-4030-4431-8334-4ac98d53c6ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:31.591748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.591721    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:31.591840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.591754    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:31.591840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.591772    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:31.592140 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.592115    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a21254-4030-4431-8334-4ac98d53c6ab-tls-certs" pod "c7a21254-4030-4431-8334-4ac98d53c6ab" (UID: "c7a21254-4030-4431-8334-4ac98d53c6ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:31.593151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.593115    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-dshm" pod "c7a21254-4030-4431-8334-4ac98d53c6ab" (UID: "c7a21254-4030-4431-8334-4ac98d53c6ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:31.593263 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.593235    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a21254-4030-4431-8334-4ac98d53c6ab-kube-api-access-pmx82" pod "c7a21254-4030-4431-8334-4ac98d53c6ab" (UID: "c7a21254-4030-4431-8334-4ac98d53c6ab"). InnerVolumeSpecName "kube-api-access-pmx82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:31.693680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.693642    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmx82\" (UniqueName: \"kubernetes.io/projected/c7a21254-4030-4431-8334-4ac98d53c6ab-kube-api-access-pmx82\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:31.693680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.693686    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7a21254-4030-4431-8334-4ac98d53c6ab-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:31.693907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:31.693698    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7a21254-4030-4431-8334-4ac98d53c6ab-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:32.334567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:32.334533    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"
Nov 03 11:21:32.334567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:32.334556    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq" event={"ID":"c7a21254-4030-4431-8334-4ac98d53c6ab","Type":"ContainerDied","Data":"df0ce1558003f26fc392d15e2f03e898e9b01ce1c158c6f4fd519975976c6a0c"}
Nov 03 11:21:32.335149 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:32.334611    5393 scope.go:122] "RemoveContainer" containerID="13912600d24687c351e91b0dc40cf371fd25f2188dc6f82e11a29197c286b1c1"
Nov 03 11:21:32.356669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:32.356638    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"]
Nov 03 11:21:32.361347 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:32.361323    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-055d82/test-model-055d82-kserve-8568995767-v5zxq"]
Nov 03 11:21:34.234121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:34.234087    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a21254-4030-4431-8334-4ac98d53c6ab" path="/var/lib/kubelet/pods/c7a21254-4030-4431-8334-4ac98d53c6ab/volumes"
Nov 03 11:21:37.485773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:37.485727    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"]
Nov 03 11:21:41.994156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:41.994111    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw" podUID="791f8324-baed-4007-975c-9995b66647f4" containerName="main" containerID="cri-o://6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c" gracePeriod=45
Nov 03 11:21:42.265384 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.265310    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"]
Nov 03 11:21:42.485513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.485118    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:21:42.492079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.491997    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252" podUID="30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" containerName="payload-pre-processing" containerID="cri-o://f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711" gracePeriod=25
Nov 03 11:21:42.576134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.575956    5393 generic.go:356] "Generic (PLEG): container finished" podID="791f8324-baed-4007-975c-9995b66647f4" containerID="6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c" exitCode=0
Nov 03 11:21:42.576456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.576074    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"
Nov 03 11:21:42.577047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.577011    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw" event={"ID":"791f8324-baed-4007-975c-9995b66647f4","Type":"ContainerDied","Data":"6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c"}
Nov 03 11:21:42.577227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.577212    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw" event={"ID":"791f8324-baed-4007-975c-9995b66647f4","Type":"ContainerDied","Data":"43a3601e6d00a7c7925beef060946e2a53e00729eb7167549008d061f5fcc31d"}
Nov 03 11:21:42.577362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.577348    5393 scope.go:122] "RemoveContainer" containerID="6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c"
Nov 03 11:21:42.606257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.606226    5393 scope.go:122] "RemoveContainer" containerID="6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c"
Nov 03 11:21:42.606657 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:21:42.606629    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c\": container with ID starting with 6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c not found: ID does not exist" containerID="6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c"
Nov 03 11:21:42.606855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.606666    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c"} err="failed to get container status \"6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c\": rpc error: code = NotFound desc = could not find container \"6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c\": container with ID starting with 6f82d6fd7611de63d7a87d0f8d2804fcdaad40c2a0cb25c6b70a45d61873f08c not found: ID does not exist"
Nov 03 11:21:42.719953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.706866    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-dshm\") pod \"791f8324-baed-4007-975c-9995b66647f4\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") "
Nov 03 11:21:42.719953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.707025    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-home\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-home\") pod \"791f8324-baed-4007-975c-9995b66647f4\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") "
Nov 03 11:21:42.719953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.707166    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/791f8324-baed-4007-975c-9995b66647f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/791f8324-baed-4007-975c-9995b66647f4-tls-certs\") pod \"791f8324-baed-4007-975c-9995b66647f4\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") "
Nov 03 11:21:42.719953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.707246    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791f8324-baed-4007-975c-9995b66647f4-kube-api-access-bsg7z\" (UniqueName: \"kubernetes.io/projected/791f8324-baed-4007-975c-9995b66647f4-kube-api-access-bsg7z\") pod \"791f8324-baed-4007-975c-9995b66647f4\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") "
Nov 03 11:21:42.719953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.707295    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-model-cache\") pod \"791f8324-baed-4007-975c-9995b66647f4\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") "
Nov 03 11:21:42.719953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.707352    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-tmp-dir\") pod \"791f8324-baed-4007-975c-9995b66647f4\" (UID: \"791f8324-baed-4007-975c-9995b66647f4\") "
Nov 03 11:21:42.768995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.756113    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-home" pod "791f8324-baed-4007-975c-9995b66647f4" (UID: "791f8324-baed-4007-975c-9995b66647f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:42.768995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.756586    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-model-cache" pod "791f8324-baed-4007-975c-9995b66647f4" (UID: "791f8324-baed-4007-975c-9995b66647f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:42.774001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.771903    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791f8324-baed-4007-975c-9995b66647f4-tls-certs" pod "791f8324-baed-4007-975c-9995b66647f4" (UID: "791f8324-baed-4007-975c-9995b66647f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:42.774001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.772249    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-tmp-dir" pod "791f8324-baed-4007-975c-9995b66647f4" (UID: "791f8324-baed-4007-975c-9995b66647f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:42.778556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.778524    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-dshm" pod "791f8324-baed-4007-975c-9995b66647f4" (UID: "791f8324-baed-4007-975c-9995b66647f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:42.782810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.782782    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791f8324-baed-4007-975c-9995b66647f4-kube-api-access-bsg7z" pod "791f8324-baed-4007-975c-9995b66647f4" (UID: "791f8324-baed-4007-975c-9995b66647f4"). InnerVolumeSpecName "kube-api-access-bsg7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:42.811785 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.811753    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791f8324-baed-4007-975c-9995b66647f4-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:42.811785 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.811787    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg7z\" (UniqueName: \"kubernetes.io/projected/791f8324-baed-4007-975c-9995b66647f4-kube-api-access-bsg7z\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:42.812021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.811801    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:42.812021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.811815    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:42.812021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.811827    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:42.812021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.811839    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791f8324-baed-4007-975c-9995b66647f4-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:42.904482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.904453    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"]
Nov 03 11:21:42.908951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:42.908908    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw"]
Nov 03 11:21:43.193080 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:21:43.192996    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:21:43.406060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:43.406016    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-24376f/test-model-24376f-kserve-6448887f87-mn8lw" podUID="791f8324-baed-4007-975c-9995b66647f4" containerName="main" probeResult="failure" output="Get \"https://10.134.1.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:44.235714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:44.235680    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791f8324-baed-4007-975c-9995b66647f4" path="/var/lib/kubelet/pods/791f8324-baed-4007-975c-9995b66647f4/volumes"
Nov 03 11:21:45.554093 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:45.554047    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252" podUID="30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:21:52.808586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:52.802972    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"]
Nov 03 11:21:52.808586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:52.803242    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" containerID="cri-o://ec512e80510fbe3e08156377a6d1eea55ad535ddfdef0a11627b5a870d1514a1" gracePeriod=30
Nov 03 11:21:53.254675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.236148    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"]
Nov 03 11:21:53.254675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.242218    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy" containerID="cri-o://ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f" gracePeriod=30
Nov 03 11:21:53.413942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.388941    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98" podUID="feaf2442-f46e-4d47-8b95-6b5465574620" containerName="main" probeResult="failure" output="Get \"https://10.134.1.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.503944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.478154    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.612074 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.612024    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg" podUID="f00dd0cb-ef27-428f-b926-21396cb38b43" containerName="main" probeResult="failure" output="Get \"https://10.134.1.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.726177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.726123    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.793956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.792822    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6" podUID="b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" containerName="main" probeResult="failure" output="Get \"https://10.134.1.30:8000/ready\": context deadline exceeded"
Nov 03 11:21:53.816947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.812010    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.924164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.922249    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.924164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.922436    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.983280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.942992    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Nov 03 11:21:53.983280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.964582    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:53.983280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:53.982620    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.035048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.020228    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm" podUID="b9b97e11-284c-483d-a6b3-a50073df6247" containerName="main" probeResult="failure" output="Get \"https://10.134.1.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.090057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.073779    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.313003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.312114    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27" podUID="ea7380ea-3664-469f-a0d4-c947bdecb710" containerName="main" probeResult="failure" output="Get \"https://10.134.1.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.313003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.312833    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.338161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.314326    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.504946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.500242    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.816113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.816061    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.874949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.853060    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Nov 03 11:21:54.931945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.906655    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"]
Nov 03 11:21:54.934233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.934210    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:21:54.934995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.934034    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.935200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.935173    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 11:21:54.935383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.935359    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 11:21:54.950989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.936309    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 11:21:54.950989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.948696    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:54.962432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.962405    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 11:21:54.972011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:54.965625    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:21:55.010947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.004347    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 11:21:55.037570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.037536    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"]
Nov 03 11:21:55.039192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.039163    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5630cb64-48b7-4aa0-8cf1-64f867be7f06" containerName="revoke-keys"
Nov 03 11:21:55.039456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.039437    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5630cb64-48b7-4aa0-8cf1-64f867be7f06" containerName="revoke-keys"
Nov 03 11:21:55.039626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.039612    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c7a21254-4030-4431-8334-4ac98d53c6ab" containerName="main"
Nov 03 11:21:55.039731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.039718    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a21254-4030-4431-8334-4ac98d53c6ab" containerName="main"
Nov 03 11:21:55.039936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.039901    5393 cpu_manager.go:414] "containerMap: removing container" podUID="791f8324-baed-4007-975c-9995b66647f4" containerName="main"
Nov 03 11:21:55.040061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.040047    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791f8324-baed-4007-975c-9995b66647f4" containerName="main"
Nov 03 11:21:55.040290 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.040256    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.040855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.040337    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a21254-4030-4431-8334-4ac98d53c6ab" containerName="main"
Nov 03 11:21:55.041038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.041021    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="791f8324-baed-4007-975c-9995b66647f4" containerName="main"
Nov 03 11:21:55.041170 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.041156    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5630cb64-48b7-4aa0-8cf1-64f867be7f06" containerName="revoke-keys"
Nov 03 11:21:55.049739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.049715    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 11:21:55.053174 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.050506    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.056763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.056725    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 11:21:55.069950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.069835    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:21:55.070239 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.070221    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:21:55.102752 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.102722    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"]
Nov 03 11:21:55.134054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.117009    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.134054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.117355    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 11:21:55.212940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.212232    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.213142 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.213080    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.236944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.213408    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ff5z\" (UniqueName: \"kubernetes.io/projected/0fb61021-2afe-421b-b2fc-16b70954d7f1-kube-api-access-8ff5z\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.293941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.293395    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 11:21:55.324117 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.319451    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/ready\": context deadline exceeded"
Nov 03 11:21:55.324117 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.320016    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:21:55.348945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.347766    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ff5z\" (UniqueName: \"kubernetes.io/projected/0fb61021-2afe-421b-b2fc-16b70954d7f1-kube-api-access-8ff5z\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.348945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.348074    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.348945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.348126    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.349293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.349123    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.392080 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.391039    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:21:55.431488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.426402    5393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 11:21:55.478988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.474042    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.495689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.483880    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 11:21:55.521942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.514574    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ff5z\" (UniqueName: \"kubernetes.io/projected/0fb61021-2afe-421b-b2fc-16b70954d7f1-kube-api-access-8ff5z\") pod \"maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.573945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.569081    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252" podUID="30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:21:55.579282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.577847    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 11:21:55.767988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.740443    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:55.788538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.785772    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:21:55.788538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:55.786617    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 11:21:56.180574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:56.179831    5393 patch_prober.go:29] interesting pod/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:21:56.180574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:56.179902    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:21:56.869824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:56.869769    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"]
Nov 03 11:21:56.876944 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:21:56.875106    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb61021_2afe_421b_b2fc_16b70954d7f1.slice/crio-6b8fbde052156182b41e74536523eb8aafc84b088c2398147fac9c3100dfd81e WatchSource:0}: Error finding container 6b8fbde052156182b41e74536523eb8aafc84b088c2398147fac9c3100dfd81e: Status 404 returned error can't find the container with id 6b8fbde052156182b41e74536523eb8aafc84b088c2398147fac9c3100dfd81e
Nov 03 11:21:57.226503 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.226471    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc" event={"ID":"0fb61021-2afe-421b-b2fc-16b70954d7f1","Type":"ContainerStarted","Data":"6b8fbde052156182b41e74536523eb8aafc84b088c2398147fac9c3100dfd81e"}
Nov 03 11:21:57.457005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.456877    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98" podUID="feaf2442-f46e-4d47-8b95-6b5465574620" containerName="main" containerID="cri-o://b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605" gracePeriod=45
Nov 03 11:21:57.805489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.805460    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:21:57.891633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.891585    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feaf2442-f46e-4d47-8b95-6b5465574620-kube-api-access-xvhtn\" (UniqueName: \"kubernetes.io/projected/feaf2442-f46e-4d47-8b95-6b5465574620-kube-api-access-xvhtn\") pod \"feaf2442-f46e-4d47-8b95-6b5465574620\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") "
Nov 03 11:21:57.891820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.891693    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-tmp-dir\") pod \"feaf2442-f46e-4d47-8b95-6b5465574620\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") "
Nov 03 11:21:57.891820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.891766    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-model-cache\") pod \"feaf2442-f46e-4d47-8b95-6b5465574620\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") "
Nov 03 11:21:57.892018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.891948    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-tmp-dir" pod "feaf2442-f46e-4d47-8b95-6b5465574620" (UID: "feaf2442-f46e-4d47-8b95-6b5465574620"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:57.892018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.891957    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-home\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-home\") pod \"feaf2442-f46e-4d47-8b95-6b5465574620\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") "
Nov 03 11:21:57.892144 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.892113    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feaf2442-f46e-4d47-8b95-6b5465574620-tls-certs\" (UniqueName: \"kubernetes.io/secret/feaf2442-f46e-4d47-8b95-6b5465574620-tls-certs\") pod \"feaf2442-f46e-4d47-8b95-6b5465574620\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") "
Nov 03 11:21:57.892248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.892107    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-model-cache" pod "feaf2442-f46e-4d47-8b95-6b5465574620" (UID: "feaf2442-f46e-4d47-8b95-6b5465574620"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:57.892317 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.892156    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-home" pod "feaf2442-f46e-4d47-8b95-6b5465574620" (UID: "feaf2442-f46e-4d47-8b95-6b5465574620"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:57.892317 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.892278    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-dshm\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-dshm\") pod \"feaf2442-f46e-4d47-8b95-6b5465574620\" (UID: \"feaf2442-f46e-4d47-8b95-6b5465574620\") "
Nov 03 11:21:57.893322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.893296    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:57.893440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.893328    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:57.893440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.893343    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:57.894233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.894212    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feaf2442-f46e-4d47-8b95-6b5465574620-kube-api-access-xvhtn" pod "feaf2442-f46e-4d47-8b95-6b5465574620" (UID: "feaf2442-f46e-4d47-8b95-6b5465574620"). InnerVolumeSpecName "kube-api-access-xvhtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:57.894504 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.894485    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feaf2442-f46e-4d47-8b95-6b5465574620-tls-certs" pod "feaf2442-f46e-4d47-8b95-6b5465574620" (UID: "feaf2442-f46e-4d47-8b95-6b5465574620"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:57.894573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.894518    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-dshm" pod "feaf2442-f46e-4d47-8b95-6b5465574620" (UID: "feaf2442-f46e-4d47-8b95-6b5465574620"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:57.996022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.995972    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feaf2442-f46e-4d47-8b95-6b5465574620-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:57.996022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.996024    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feaf2442-f46e-4d47-8b95-6b5465574620-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:57.996234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:57.996040    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvhtn\" (UniqueName: \"kubernetes.io/projected/feaf2442-f46e-4d47-8b95-6b5465574620-kube-api-access-xvhtn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.251802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.251738    5393 generic.go:356] "Generic (PLEG): container finished" podID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerID="ec512e80510fbe3e08156377a6d1eea55ad535ddfdef0a11627b5a870d1514a1" exitCode=0
Nov 03 11:21:58.252287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.251820    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" event={"ID":"7086125d-88b6-4b83-8e78-9cc56eadec7e","Type":"ContainerDied","Data":"ec512e80510fbe3e08156377a6d1eea55ad535ddfdef0a11627b5a870d1514a1"}
Nov 03 11:21:58.256965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.255206    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc" event={"ID":"0fb61021-2afe-421b-b2fc-16b70954d7f1","Type":"ContainerStarted","Data":"b02c5a298480496f2bcffa0fb7c31d67ef899b37447ef435e42031fe29810de6"}
Nov 03 11:21:58.256965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.256778    5393 generic.go:356] "Generic (PLEG): container finished" podID="feaf2442-f46e-4d47-8b95-6b5465574620" containerID="b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605" exitCode=0
Nov 03 11:21:58.256965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.256825    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"
Nov 03 11:21:58.256965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.256856    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98" event={"ID":"feaf2442-f46e-4d47-8b95-6b5465574620","Type":"ContainerDied","Data":"b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605"}
Nov 03 11:21:58.256965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.256900    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98" event={"ID":"feaf2442-f46e-4d47-8b95-6b5465574620","Type":"ContainerDied","Data":"f171136623e658c022dd515bff50004a07f3445493b65d86f4dfe13763c52faa"}
Nov 03 11:21:58.256965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.256940    5393 scope.go:122] "RemoveContainer" containerID="b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605"
Nov 03 11:21:58.300013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.299985    5393 scope.go:122] "RemoveContainer" containerID="b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605"
Nov 03 11:21:58.300485 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:21:58.300452    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605\": container with ID starting with b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605 not found: ID does not exist" containerID="b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605"
Nov 03 11:21:58.300608 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.300499    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605"} err="failed to get container status \"b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605\": rpc error: code = NotFound desc = could not find container \"b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605\": container with ID starting with b88f687163ec21221378d95e4aadf884b3f7634dfacb8d566d0c35a79d396605 not found: ID does not exist"
Nov 03 11:21:58.310971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.310912    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"]
Nov 03 11:21:58.315594 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.315565    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a68103/test-model-a68103-kserve-6f859fbd9c-gdx98"]
Nov 03 11:21:58.445045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.445023    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:21:58.505660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.505610    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-certs\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.505869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.505701    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-credential-socket\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.505869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.505825    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-data\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.506052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.505902    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7086125d-88b6-4b83-8e78-9cc56eadec7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7086125d-88b6-4b83-8e78-9cc56eadec7e-istiod-ca-cert\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.506052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.505977    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-certs" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:58.506052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.506025    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-token\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-token\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.506232 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.506081    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-envoy\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.506232 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.506196    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-data" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:58.506232 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.506215    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-credential-socket" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:58.506460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.506436    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-podinfo\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.506559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.506490    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7086125d-88b6-4b83-8e78-9cc56eadec7e-istiod-ca-cert" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:58.506559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.506526    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-socket\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.506686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.506568    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-kube-api-access-sj7mc\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-kube-api-access-sj7mc\") pod \"7086125d-88b6-4b83-8e78-9cc56eadec7e\" (UID: \"7086125d-88b6-4b83-8e78-9cc56eadec7e\") "
Nov 03 11:21:58.507147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.507097    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-socket" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:58.507707 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.507682    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.507824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.507711    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.507824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.507727    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.507824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.507741    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.507824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.507755    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7086125d-88b6-4b83-8e78-9cc56eadec7e-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.509136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.509110    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-podinfo" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:21:58.509347 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.509319    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-envoy" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:58.509454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.509350    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-token" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:58.509846 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.509824    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-kube-api-access-sj7mc" pod "7086125d-88b6-4b83-8e78-9cc56eadec7e" (UID: "7086125d-88b6-4b83-8e78-9cc56eadec7e"). InnerVolumeSpecName "kube-api-access-sj7mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:58.610041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.610004    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj7mc\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-kube-api-access-sj7mc\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.610041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.610037    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.610274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.610052    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:58.610274 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:58.610066    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7086125d-88b6-4b83-8e78-9cc56eadec7e-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.267626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.267599    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"
Nov 03 11:21:59.268123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.267594    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql" event={"ID":"7086125d-88b6-4b83-8e78-9cc56eadec7e","Type":"ContainerDied","Data":"23ebc60c637ac5a85e18f1f0df159a6e68174e745a0c24eccaa27de51631338c"}
Nov 03 11:21:59.268123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.267757    5393 scope.go:122] "RemoveContainer" containerID="ec512e80510fbe3e08156377a6d1eea55ad535ddfdef0a11627b5a870d1514a1"
Nov 03 11:21:59.269693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.269669    5393 generic.go:356] "Generic (PLEG): container finished" podID="0fb61021-2afe-421b-b2fc-16b70954d7f1" containerID="b02c5a298480496f2bcffa0fb7c31d67ef899b37447ef435e42031fe29810de6" exitCode=0
Nov 03 11:21:59.269817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.269751    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc" event={"ID":"0fb61021-2afe-421b-b2fc-16b70954d7f1","Type":"ContainerDied","Data":"b02c5a298480496f2bcffa0fb7c31d67ef899b37447ef435e42031fe29810de6"}
Nov 03 11:21:59.337380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.337346    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"]
Nov 03 11:21:59.344975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.343502    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-jbsql"]
Nov 03 11:21:59.559877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.559848    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:21:59.589903 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.589872    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:21:59.641058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.641024    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb61021-2afe-421b-b2fc-16b70954d7f1-kube-api-access-8ff5z\" (UniqueName: \"kubernetes.io/projected/0fb61021-2afe-421b-b2fc-16b70954d7f1-kube-api-access-8ff5z\") pod \"0fb61021-2afe-421b-b2fc-16b70954d7f1\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") "
Nov 03 11:21:59.641278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.641190    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-xks-ca\") pod \"0fb61021-2afe-421b-b2fc-16b70954d7f1\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") "
Nov 03 11:21:59.641372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.641300    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-openshift-ca\") pod \"0fb61021-2afe-421b-b2fc-16b70954d7f1\" (UID: \"0fb61021-2afe-421b-b2fc-16b70954d7f1\") "
Nov 03 11:21:59.642811 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.642779    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-openshift-ca" pod "0fb61021-2afe-421b-b2fc-16b70954d7f1" (UID: "0fb61021-2afe-421b-b2fc-16b70954d7f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:59.646030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.645997    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-xks-ca" pod "0fb61021-2afe-421b-b2fc-16b70954d7f1" (UID: "0fb61021-2afe-421b-b2fc-16b70954d7f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:21:59.648325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.648302    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb61021-2afe-421b-b2fc-16b70954d7f1-kube-api-access-8ff5z" pod "0fb61021-2afe-421b-b2fc-16b70954d7f1" (UID: "0fb61021-2afe-421b-b2fc-16b70954d7f1"). InnerVolumeSpecName "kube-api-access-8ff5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:59.743246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743215    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-kube-api-access-w6gpt\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-kube-api-access-w6gpt\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743334    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-istio-token\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-istio-token\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743376    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-envoy\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743395    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-socket\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743478    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-data\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743518    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-certs\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743554    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-credential-socket\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743673    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2310479c-98fb-4cc8-8b03-59003bd96943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2310479c-98fb-4cc8-8b03-59003bd96943-istio-podinfo\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743730    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2310479c-98fb-4cc8-8b03-59003bd96943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2310479c-98fb-4cc8-8b03-59003bd96943-istiod-ca-cert\") pod \"2310479c-98fb-4cc8-8b03-59003bd96943\" (UID: \"2310479c-98fb-4cc8-8b03-59003bd96943\") "
Nov 03 11:21:59.743796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743770    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-socket" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:59.743796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743784    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-data" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:59.744060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.743969    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-credential-socket" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:59.744269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744205    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2310479c-98fb-4cc8-8b03-59003bd96943-istiod-ca-cert" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:21:59.744269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744157    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-certs" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:59.744743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744724    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.744743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744743    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.744873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744752    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.744873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744762    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb61021-2afe-421b-b2fc-16b70954d7f1-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.744873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744770    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.744873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744780    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.744873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744788    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2310479c-98fb-4cc8-8b03-59003bd96943-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.744873 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.744798    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ff5z\" (UniqueName: \"kubernetes.io/projected/0fb61021-2afe-421b-b2fc-16b70954d7f1-kube-api-access-8ff5z\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.745695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.745676    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-kube-api-access-w6gpt" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "kube-api-access-w6gpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:59.745933 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.745900    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-istio-token" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:21:59.746183 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.746162    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-envoy" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:21:59.746419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.746395    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2310479c-98fb-4cc8-8b03-59003bd96943-istio-podinfo" pod "2310479c-98fb-4cc8-8b03-59003bd96943" (UID: "2310479c-98fb-4cc8-8b03-59003bd96943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:21:59.863731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.863689    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2310479c-98fb-4cc8-8b03-59003bd96943-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.863731 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.863735    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6gpt\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-kube-api-access-w6gpt\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.864146 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.863765    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2310479c-98fb-4cc8-8b03-59003bd96943-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:21:59.864146 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:21:59.863781    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2310479c-98fb-4cc8-8b03-59003bd96943-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:00.239377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.239299    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" path="/var/lib/kubelet/pods/7086125d-88b6-4b83-8e78-9cc56eadec7e/volumes"
Nov 03 11:22:00.239863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.239848    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feaf2442-f46e-4d47-8b95-6b5465574620" path="/var/lib/kubelet/pods/feaf2442-f46e-4d47-8b95-6b5465574620/volumes"
Nov 03 11:22:00.283083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.283054    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"
Nov 03 11:22:00.283546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.283085    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc" event={"ID":"0fb61021-2afe-421b-b2fc-16b70954d7f1","Type":"ContainerDied","Data":"6b8fbde052156182b41e74536523eb8aafc84b088c2398147fac9c3100dfd81e"}
Nov 03 11:22:00.283546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.283143    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b8fbde052156182b41e74536523eb8aafc84b088c2398147fac9c3100dfd81e"
Nov 03 11:22:00.285213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.285190    5393 generic.go:356] "Generic (PLEG): container finished" podID="2310479c-98fb-4cc8-8b03-59003bd96943" containerID="ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f" exitCode=0
Nov 03 11:22:00.285635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.285277    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" event={"ID":"2310479c-98fb-4cc8-8b03-59003bd96943","Type":"ContainerDied","Data":"ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f"}
Nov 03 11:22:00.285635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.285318    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k" event={"ID":"2310479c-98fb-4cc8-8b03-59003bd96943","Type":"ContainerDied","Data":"8514c96afbb0fda37d833d3f4f8e63de2ef31d06e71e4be6522e042490a1fce3"}
Nov 03 11:22:00.285635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.285348    5393 scope.go:122] "RemoveContainer" containerID="ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f"
Nov 03 11:22:00.285635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.285349    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"
Nov 03 11:22:00.303829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.303797    5393 scope.go:122] "RemoveContainer" containerID="64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d"
Nov 03 11:22:00.309621 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.309589    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"]
Nov 03 11:22:00.314751 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.314725    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-055d82-openshift-default-79544dc787-bln7k"]
Nov 03 11:22:00.317814 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.317787    5393 scope.go:122] "RemoveContainer" containerID="ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f"
Nov 03 11:22:00.318188 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:00.318168    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f\": container with ID starting with ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f not found: ID does not exist" containerID="ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f"
Nov 03 11:22:00.318295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.318203    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f"} err="failed to get container status \"ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f\": rpc error: code = NotFound desc = could not find container \"ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f\": container with ID starting with ec8c700fd0647214d65038a24d4572a5905c0f6d750372bd789b56efa326937f not found: ID does not exist"
Nov 03 11:22:00.318295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.318227    5393 scope.go:122] "RemoveContainer" containerID="64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d"
Nov 03 11:22:00.318518 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:00.318493    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d\": container with ID starting with 64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d not found: ID does not exist" containerID="64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d"
Nov 03 11:22:00.318563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:00.318526    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d"} err="failed to get container status \"64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d\": rpc error: code = NotFound desc = could not find container \"64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d\": container with ID starting with 64c9e55f4ddfd108b2f587b75738bb71bad167b0f4edb33e7a3ab394fd7cd78d not found: ID does not exist"
Nov 03 11:22:01.137166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:01.137133    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"]
Nov 03 11:22:01.137384 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:01.137336    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy" containerID="cri-o://4cd6dd54a0f52d65b7b541c676b7d290fb138a306f1385163dc0f2dc8f426fc9" gracePeriod=30
Nov 03 11:22:02.233683 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:02.233645    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" path="/var/lib/kubelet/pods/2310479c-98fb-4cc8-8b03-59003bd96943/volumes"
Nov 03 11:22:02.740964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:02.740911    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:22:02.820815 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:02.820713    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-plugins-config-volume\") pod \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\" (UID: \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\") "
Nov 03 11:22:02.821038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:02.820882    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-kube-api-access-n8xns\" (UniqueName: \"kubernetes.io/projected/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-kube-api-access-n8xns\") pod \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\" (UID: \"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024\") "
Nov 03 11:22:02.821207 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:02.821178    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-plugins-config-volume" pod "30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" (UID: "30acda02-b9e6-4bdf-a3b4-5f82b0ef4024"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:02.821798 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:02.821752    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:02.823282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:02.823249    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-kube-api-access-n8xns" pod "30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" (UID: "30acda02-b9e6-4bdf-a3b4-5f82b0ef4024"). InnerVolumeSpecName "kube-api-access-n8xns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:02.923289 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:02.923259    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8xns\" (UniqueName: \"kubernetes.io/projected/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024-kube-api-access-n8xns\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:03.316278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.316211    5393 generic.go:356] "Generic (PLEG): container finished" podID="30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" containerID="f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711" exitCode=0
Nov 03 11:22:03.316278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.316264    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"
Nov 03 11:22:03.316787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.316303    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252" event={"ID":"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024","Type":"ContainerDied","Data":"f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711"}
Nov 03 11:22:03.316787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.316362    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252" event={"ID":"30acda02-b9e6-4bdf-a3b4-5f82b0ef4024","Type":"ContainerDied","Data":"efba5850d74bfd3a7f7bc6d4665b1f46490ba19d4675b4f72454e5c3afa910f2"}
Nov 03 11:22:03.316787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.316387    5393 scope.go:122] "RemoveContainer" containerID="f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711"
Nov 03 11:22:03.328736 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.328719    5393 scope.go:122] "RemoveContainer" containerID="f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711"
Nov 03 11:22:03.329019 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:03.328998    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711\": container with ID starting with f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711 not found: ID does not exist" containerID="f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711"
Nov 03 11:22:03.329085 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.329032    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711"} err="failed to get container status \"f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711\": rpc error: code = NotFound desc = could not find container \"f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711\": container with ID starting with f20b5b6960731e454d0ca9cd69370abeefd49d9d3bcabb86464876dbd49f6711 not found: ID does not exist"
Nov 03 11:22:03.343166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.343138    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"]
Nov 03 11:22:03.352141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:03.352106    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-055d82-5d88457c99-w5252"]
Nov 03 11:22:04.233433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:04.233396    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" path="/var/lib/kubelet/pods/30acda02-b9e6-4bdf-a3b4-5f82b0ef4024/volumes"
Nov 03 11:22:05.059055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:05.058310    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"]
Nov 03 11:22:05.059055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:05.058595    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy" containerID="cri-o://17805146d7a56bbbef61c7748e8442fd3149a6f7e94b931a9666291d110b6f7d" gracePeriod=30
Nov 03 11:22:05.203660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:05.203627    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"]
Nov 03 11:22:06.418097 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.417908    5393 generic.go:356] "Generic (PLEG): container finished" podID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerID="4cd6dd54a0f52d65b7b541c676b7d290fb138a306f1385163dc0f2dc8f426fc9" exitCode=0
Nov 03 11:22:06.418097 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.418060    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" event={"ID":"648d3e6d-2f48-4a99-b293-9b124b51ebf1","Type":"ContainerDied","Data":"4cd6dd54a0f52d65b7b541c676b7d290fb138a306f1385163dc0f2dc8f426fc9"}
Nov 03 11:22:06.418623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.418185    5393 scope.go:122] "RemoveContainer" containerID="f3f4d76edef6865ea7eb4859dbc6c742b3355bcbd6ff9bb5f15530cc20c4d974"
Nov 03 11:22:06.473126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.473093    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521182    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-certs\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521348    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-envoy\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521451    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istiod-ca-cert\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521487    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-credential-socket\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521517    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-socket\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521561    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-podinfo\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521614    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-kube-api-access-vh242\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-kube-api-access-vh242\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521686    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-token\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-token\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.521785    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-data\") pod \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\" (UID: \"648d3e6d-2f48-4a99-b293-9b124b51ebf1\") "
Nov 03 11:22:06.523808 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.523783    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-data" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:06.524171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.524150    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-certs" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:06.525481 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.525458    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-socket" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:06.525946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.525900    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istiod-ca-cert" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:06.526349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.526328    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-credential-socket" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:06.544343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.536012    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-podinfo" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:06.544343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.536214    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-kube-api-access-vh242" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "kube-api-access-vh242". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:06.544343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.538329    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-token" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:06.556074 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.556046    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-envoy" pod "648d3e6d-2f48-4a99-b293-9b124b51ebf1" (UID: "648d3e6d-2f48-4a99-b293-9b124b51ebf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.746278    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.783521    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.783538    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.783550    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.783562    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.783574    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh242\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-kube-api-access-vh242\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.783588    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.783600    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:06.791005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:06.783612    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648d3e6d-2f48-4a99-b293-9b124b51ebf1-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:07.020943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:07.015889    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"]
Nov 03 11:22:07.026493 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:07.026403    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a68103-6bqsc"]
Nov 03 11:22:07.429597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:07.429559    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"
Nov 03 11:22:07.430131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:07.429593    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m" event={"ID":"648d3e6d-2f48-4a99-b293-9b124b51ebf1","Type":"ContainerDied","Data":"0e17babdb8d22b07dd28770d391b5b2c85d4cbe9de6eb7fb63934406edc0052f"}
Nov 03 11:22:07.430131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:07.429655    5393 scope.go:122] "RemoveContainer" containerID="4cd6dd54a0f52d65b7b541c676b7d290fb138a306f1385163dc0f2dc8f426fc9"
Nov 03 11:22:07.464812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:07.464769    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"]
Nov 03 11:22:07.469581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:07.469555    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5d3c3-openshift-default-67cf884fd9-jsg2m"]
Nov 03 11:22:08.234071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:08.234028    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb61021-2afe-421b-b2fc-16b70954d7f1" path="/var/lib/kubelet/pods/0fb61021-2afe-421b-b2fc-16b70954d7f1/volumes"
Nov 03 11:22:08.234626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:08.234601    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" path="/var/lib/kubelet/pods/648d3e6d-2f48-4a99-b293-9b124b51ebf1/volumes"
Nov 03 11:22:08.572795 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:08.572705    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"]
Nov 03 11:22:08.573307 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:08.572994    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy" containerID="cri-o://3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae" gracePeriod=30
Nov 03 11:22:10.029049 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.026548    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6" podUID="b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" containerName="main" containerID="cri-o://69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c" gracePeriod=45
Nov 03 11:22:10.204962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.204888    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc" podUID="ed16f6d6-d892-4603-a936-40f5e7ca4e73" containerName="payload-pre-processing" containerID="cri-o://48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7" gracePeriod=25
Nov 03 11:22:10.426454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.426427    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:22:10.460829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.460793    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-model-cache\") pod \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") "
Nov 03 11:22:10.461031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.460943    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-kube-api-access-lffgz\" (UniqueName: \"kubernetes.io/projected/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-kube-api-access-lffgz\") pod \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") "
Nov 03 11:22:10.461031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.460988    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tls-certs\" (UniqueName: \"kubernetes.io/secret/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tls-certs\") pod \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") "
Nov 03 11:22:10.461330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.461309    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-dshm\") pod \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") "
Nov 03 11:22:10.461525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.461509    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tmp-dir\") pod \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") "
Nov 03 11:22:10.461593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.461551    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-home\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-home\") pod \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\" (UID: \"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057\") "
Nov 03 11:22:10.462846 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.462660    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-model-cache" pod "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" (UID: "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.463089 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.462843    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tmp-dir" pod "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" (UID: "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.463377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.463352    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-home" pod "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" (UID: "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.464886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.464862    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tls-certs" pod "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" (UID: "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:10.465031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.465011    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-dshm" pod "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" (UID: "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.465513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.465485    5393 generic.go:356] "Generic (PLEG): container finished" podID="b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" containerID="69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c" exitCode=0
Nov 03 11:22:10.465602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.465589    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"
Nov 03 11:22:10.465682 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.465657    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-kube-api-access-lffgz" pod "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" (UID: "b974d9d2-1bb1-41ad-bc3c-872ae1e2c057"). InnerVolumeSpecName "kube-api-access-lffgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:10.465759 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.465704    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6" event={"ID":"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057","Type":"ContainerDied","Data":"69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c"}
Nov 03 11:22:10.465812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.465756    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6" event={"ID":"b974d9d2-1bb1-41ad-bc3c-872ae1e2c057","Type":"ContainerDied","Data":"4c7dda5ada8604becfbddb558a8f27031d59dc2711c4e356704f917597de554d"}
Nov 03 11:22:10.465812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.465780    5393 scope.go:122] "RemoveContainer" containerID="69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c"
Nov 03 11:22:10.467522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.467501    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lffgz\" (UniqueName: \"kubernetes.io/projected/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-kube-api-access-lffgz\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.467620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.467529    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.467620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.467544    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.467620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.467580    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.467620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.467593    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.467620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.467607    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.469676 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.469645    5393 generic.go:356] "Generic (PLEG): container finished" podID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerID="17805146d7a56bbbef61c7748e8442fd3149a6f7e94b931a9666291d110b6f7d" exitCode=0
Nov 03 11:22:10.469801 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.469777    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" event={"ID":"2bb2139c-311d-4d53-bebf-cf7c29dd567e","Type":"ContainerDied","Data":"17805146d7a56bbbef61c7748e8442fd3149a6f7e94b931a9666291d110b6f7d"}
Nov 03 11:22:10.469860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.469839    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t" event={"ID":"2bb2139c-311d-4d53-bebf-cf7c29dd567e","Type":"ContainerDied","Data":"2b9c6f9bbdacacc47a5f5ec06dbf8b8af99b6f0df043294e7cba14bfc031f348"}
Nov 03 11:22:10.469988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.469864    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b9c6f9bbdacacc47a5f5ec06dbf8b8af99b6f0df043294e7cba14bfc031f348"
Nov 03 11:22:10.474430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.474410    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:22:10.482808 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.482789    5393 scope.go:122] "RemoveContainer" containerID="69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c"
Nov 03 11:22:10.483161 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:10.483140    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c\": container with ID starting with 69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c not found: ID does not exist" containerID="69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c"
Nov 03 11:22:10.483230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.483167    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c"} err="failed to get container status \"69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c\": rpc error: code = NotFound desc = could not find container \"69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c\": container with ID starting with 69dad43fca3a5c1d9542d40f598859badc461e8bff0dc96f782464f5f7dc506c not found: ID does not exist"
Nov 03 11:22:10.568727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.568695    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-podinfo\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.568972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.568762    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-kube-api-access-d8bg4\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-kube-api-access-d8bg4\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.568972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.568887    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-certs\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.569134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.568995    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-data\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.569134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569046    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-envoy\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.569134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569113    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-socket\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.569306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569176    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istiod-ca-cert\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.569306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569216    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-certs" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.569306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569246    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-data" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.569466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569337    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-token\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-token\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.569466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569425    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-credential-socket\") pod \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\" (UID: \"2bb2139c-311d-4d53-bebf-cf7c29dd567e\") "
Nov 03 11:22:10.569466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569439    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-socket" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.569828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569548    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istiod-ca-cert" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:10.569828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.569658    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-credential-socket" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.570202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.570181    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.570202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.570203    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.570374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.570213    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.570374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.570221    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.570374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.570231    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.571495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.571474    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-kube-api-access-d8bg4" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "kube-api-access-d8bg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:10.571584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.571514    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-token" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:10.571687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.571672    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-envoy" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:10.571761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.571693    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-podinfo" pod "2bb2139c-311d-4d53-bebf-cf7c29dd567e" (UID: "2bb2139c-311d-4d53-bebf-cf7c29dd567e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:10.685012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.681830    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.685012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.681876    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.685012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.681892    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bg4\" (UniqueName: \"kubernetes.io/projected/2bb2139c-311d-4d53-bebf-cf7c29dd567e-kube-api-access-d8bg4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.685012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.681908    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb2139c-311d-4d53-bebf-cf7c29dd567e-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:10.794170 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.794134    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"]
Nov 03 11:22:10.798642 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:10.798616    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3eb43f/test-model-3eb43f-kserve-59d588c649-269d6"]
Nov 03 11:22:11.479826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:11.479795    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"
Nov 03 11:22:11.505663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:11.505631    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"]
Nov 03 11:22:11.510824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:11.510795    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-br72t"]
Nov 03 11:22:12.260949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.248243    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" path="/var/lib/kubelet/pods/2bb2139c-311d-4d53-bebf-cf7c29dd567e/volumes"
Nov 03 11:22:12.260949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.249097    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" path="/var/lib/kubelet/pods/b974d9d2-1bb1-41ad-bc3c-872ae1e2c057/volumes"
Nov 03 11:22:12.757278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.757243    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"]
Nov 03 11:22:12.758139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758120    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy"
Nov 03 11:22:12.758139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758139    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758151    5393 cpu_manager.go:414] "containerMap: removing container" podUID="30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" containerName="payload-pre-processing"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758156    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" containerName="payload-pre-processing"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758174    5393 cpu_manager.go:414] "containerMap: removing container" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758179    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758187    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758191    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758200    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758205    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758223    5393 cpu_manager.go:414] "containerMap: removing container" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy"
Nov 03 11:22:12.758225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758227    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758253    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" containerName="main"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758257    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" containerName="main"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758270    5393 cpu_manager.go:414] "containerMap: removing container" podUID="feaf2442-f46e-4d47-8b95-6b5465574620" containerName="main"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758274    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feaf2442-f46e-4d47-8b95-6b5465574620" containerName="main"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758283    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0fb61021-2afe-421b-b2fc-16b70954d7f1" containerName="revoke-keys"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758287    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb61021-2afe-421b-b2fc-16b70954d7f1" containerName="revoke-keys"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758401    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb2139c-311d-4d53-bebf-cf7c29dd567e" containerName="istio-proxy"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758410    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="30acda02-b9e6-4bdf-a3b4-5f82b0ef4024" containerName="payload-pre-processing"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758419    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758426    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758431    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7086125d-88b6-4b83-8e78-9cc56eadec7e" containerName="istio-proxy"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758440    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="648d3e6d-2f48-4a99-b293-9b124b51ebf1" containerName="istio-proxy"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758450    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b974d9d2-1bb1-41ad-bc3c-872ae1e2c057" containerName="main"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758458    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="feaf2442-f46e-4d47-8b95-6b5465574620" containerName="main"
Nov 03 11:22:12.758532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.758464    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb61021-2afe-421b-b2fc-16b70954d7f1" containerName="revoke-keys"
Nov 03 11:22:12.763830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.763809    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.772935 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.772890    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c0a748-openshift-default-dockercfg-q6b72\""
Nov 03 11:22:12.776971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.775778    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"]
Nov 03 11:22:12.777124 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.777022    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy"
Nov 03 11:22:12.777124 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.777040    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy"
Nov 03 11:22:12.777364 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.777327    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2310479c-98fb-4cc8-8b03-59003bd96943" containerName="istio-proxy"
Nov 03 11:22:12.808158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.808128    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"]
Nov 03 11:22:12.808358 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.808307    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.814334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.814305    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"]
Nov 03 11:22:12.816786 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.816739    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.816993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.816847    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.816993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.816895    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqzl\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-kube-api-access-jgqzl\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.817209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.816996    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.817209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.817042    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.817336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.817229    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.817336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.817262    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.817336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.817308    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.817497 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.817380    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.919494    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.919549    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.919640    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.919704    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.919786    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.919909    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.919986    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920014    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920057    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5f9\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-kube-api-access-bl5f9\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920147    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920206    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqzl\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-kube-api-access-jgqzl\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920247    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920281    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920300    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920422    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920533    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.920593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920604    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.922257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920715    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.922257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920742    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.922257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920800    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:12.922257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.920985    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.922257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.921365    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.926976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.923045    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.926976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.923238    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.929815 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.929792    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.930165 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.930146    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqzl\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-kube-api-access-jgqzl\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:12.930477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:12.930452    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:13.022758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.022676    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.022758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.022713    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.022758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.022743    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.022778    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.022945    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023000    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023034    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5f9\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-kube-api-access-bl5f9\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023138    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023138    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-credential-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023146    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-socket\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023243    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023335    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-data\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023539    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-certs\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.023714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.023616    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istiod-ca-cert\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.025643 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.025619    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-envoy\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.025720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.025706    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-podinfo\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.030724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.030696    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-token\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.031110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.031087    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5f9\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-kube-api-access-bl5f9\") pod \"e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") " pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.130026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.129984    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:13.133250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.133209    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:13.197788 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:13.197731    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:22:13.430258 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:22:13.430226    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b77182d_ef2a_43e8_a7d8_3c5a790bccd6.slice/crio-d886594a6401fdf33cb7e104f13bc4216f13e202b1e0510f4b67003761ecd429 WatchSource:0}: Error finding container d886594a6401fdf33cb7e104f13bc4216f13e202b1e0510f4b67003761ecd429: Status 404 returned error can't find the container with id d886594a6401fdf33cb7e104f13bc4216f13e202b1e0510f4b67003761ecd429
Nov 03 11:22:13.430609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.430590    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"]
Nov 03 11:22:13.433699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.433666    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:13.433824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.433758    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:13.433824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.433806    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:13.435531 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:22:13.435504    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f66eaa5_f85d_4a53_88c6_c3fc84a676e4.slice/crio-49719efc81407424141d41f3284528b137687793529021857815673f1f246626 WatchSource:0}: Error finding container 49719efc81407424141d41f3284528b137687793529021857815673f1f246626: Status 404 returned error can't find the container with id 49719efc81407424141d41f3284528b137687793529021857815673f1f246626
Nov 03 11:22:13.436590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.436569    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"]
Nov 03 11:22:13.438363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.438332    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:13.438460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.438405    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:13.438521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.438470    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:22:13.527906 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.527862    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" event={"ID":"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6","Type":"ContainerStarted","Data":"d886594a6401fdf33cb7e104f13bc4216f13e202b1e0510f4b67003761ecd429"}
Nov 03 11:22:13.530379 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.530349    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" event={"ID":"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4","Type":"ContainerStarted","Data":"49719efc81407424141d41f3284528b137687793529021857815673f1f246626"}
Nov 03 11:22:13.892120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.892088    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.960650    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-envoy\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.960705    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istiod-ca-cert\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.960759    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-certs\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.960821    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-credential-socket\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.960953    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-podinfo\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.961026    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-socket\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.961147    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-data\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.961218    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-token\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-token\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.961319    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-kube-api-access-sfzp9\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-kube-api-access-sfzp9\") pod \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\" (UID: \"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48\") "
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.968542    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-certs" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.968850    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istiod-ca-cert" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.969128    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-credential-socket" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.969323    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-data" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.969578    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-socket" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:13.975388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.969671    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-envoy" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:14.001345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.977997    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-podinfo" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:14.001345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:13.983135    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-token" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:14.018438 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.015214    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-kube-api-access-sfzp9" pod "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" (UID: "db52f8d3-88ff-4cd9-91fc-9ac95ad82c48"). InnerVolumeSpecName "kube-api-access-sfzp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:14.064649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064604    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.064649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064652    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.064880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064667    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.064880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064683    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfzp9\" (UniqueName: \"kubernetes.io/projected/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-kube-api-access-sfzp9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.064880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064724    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.064880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064738    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.064880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064751    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.064880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064763    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.064880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.064779    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:14.452942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.446174    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"]
Nov 03 11:22:14.452942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.446891    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy" containerID="cri-o://6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96" gracePeriod=30
Nov 03 11:22:15.004942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:14.999183    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc" podUID="ed16f6d6-d892-4603-a936-40f5e7ca4e73" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:22:15.009626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.007530    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" event={"ID":"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4","Type":"ContainerStarted","Data":"0f4f0e8250b3dbabad9cbc47857acf47799aafe5e308bf06e54479f36d933be4"}
Nov 03 11:22:15.033011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.015955    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" event={"ID":"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6","Type":"ContainerStarted","Data":"faf88e517aa6c7af871e0cf7ace0817e08152da08b83df45b0133c8a9618fa3f"}
Nov 03 11:22:15.033011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.021344    5393 generic.go:356] "Generic (PLEG): container finished" podID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerID="3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae" exitCode=0
Nov 03 11:22:15.033011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.021499    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" event={"ID":"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48","Type":"ContainerDied","Data":"3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae"}
Nov 03 11:22:15.033011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.021558    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm" event={"ID":"db52f8d3-88ff-4cd9-91fc-9ac95ad82c48","Type":"ContainerDied","Data":"a9383e3189bf8bc5c35bc70a885c87d651bee3a4beb5ae199684fc5732191b9b"}
Nov 03 11:22:15.033011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.021591    5393 scope.go:122] "RemoveContainer" containerID="3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae"
Nov 03 11:22:15.033011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.021778    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"
Nov 03 11:22:15.089942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.085650    5393 scope.go:122] "RemoveContainer" containerID="3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae"
Nov 03 11:22:15.089942 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:15.086195    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae\": container with ID starting with 3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae not found: ID does not exist" containerID="3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae"
Nov 03 11:22:15.089942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.086253    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae"} err="failed to get container status \"3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae\": rpc error: code = NotFound desc = could not find container \"3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae\": container with ID starting with 3633e197bac3dfe9810567d9333d2a5f3e8defd38a04719c8604050e4f9aa0ae not found: ID does not exist"
Nov 03 11:22:15.148944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.148726    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:15.152942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.150291    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:15.152942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.150400    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 11:22:15.152942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.150442    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 11:22:15.176820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.174606    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 11:22:15.176820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.174662    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 11:22:15.239956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.235966    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" podStartSLOduration=3.182902698 podStartE2EDuration="3.182902698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:22:15.037656696 +0000 UTC m=+3178.942511140" watchObservedRunningTime="2026-11-03 11:22:15.182902698 +0000 UTC m=+3179.087757142"
Nov 03 11:22:15.244832 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.244762    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" podStartSLOduration=3.244746943 podStartE2EDuration="3.244746943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:22:15.081499285 +0000 UTC m=+3178.986353727" watchObservedRunningTime="2026-11-03 11:22:15.244746943 +0000 UTC m=+3179.149601384"
Nov 03 11:22:15.253034 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.253011    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"]
Nov 03 11:22:15.258017 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:15.257945    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-cbeebd-openshift-default-5ff84cbf65-8ndxm"]
Nov 03 11:22:16.133941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:16.133658    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 11:22:16.133941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:16.133729    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 11:22:16.134534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:16.134241    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 11:22:16.134534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:16.134310    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 11:22:16.256657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:16.256624    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" path="/var/lib/kubelet/pods/db52f8d3-88ff-4cd9-91fc-9ac95ad82c48/volumes"
Nov 03 11:22:17.135515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:17.135438    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 11:22:17.136199 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:17.135529    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 11:22:17.137640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:17.136483    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 11:22:17.137640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:17.136541    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 11:22:18.130944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:18.130726    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 11:22:18.130944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:18.130801    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 11:22:18.139025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:18.137301    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 11:22:18.139025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:18.137375    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 11:22:19.130459 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:19.130420    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 11:22:19.137602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:19.137069    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 11:22:19.137602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:19.137254    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 11:22:19.137602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:19.137307    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 11:22:20.014946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:19.983557    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:22:20.062183 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.062144    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istiod-ca-cert\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.062498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.062472    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-podinfo\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.062735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.062707    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-kube-api-access-dtxw6\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-kube-api-access-dtxw6\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.062902 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.062884    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-credential-socket\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.063112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.063095    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-socket\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.063342 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.063323    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-data\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.063536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.063517    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-certs\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.063726 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.063709    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-envoy\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.064031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.064011    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-token\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-token\") pod \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\" (UID: \"fccacc68-1e07-47c4-9878-6bf1dcafbd5b\") "
Nov 03 11:22:20.064747 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.064436    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-certs" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.065333 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.065267    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istiod-ca-cert" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:20.065721 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.065641    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-credential-socket" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.066039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.065931    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-socket" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.066039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.065971    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.066039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.065992    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.066239 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.066114    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-data" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.068063 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.068028    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-podinfo" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:20.075557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.075405    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-kube-api-access-dtxw6" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "kube-api-access-dtxw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:20.075557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.075476    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-envoy" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:20.075557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.075534    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-token" pod "fccacc68-1e07-47c4-9878-6bf1dcafbd5b" (UID: "fccacc68-1e07-47c4-9878-6bf1dcafbd5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:20.142068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.135355    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 11:22:20.142068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.135435    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 11:22:20.143544 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.143475    5393 patch_prober.go:29] interesting pod/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 11:22:20.143854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.143778    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 11:22:20.157879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.157834    5393 generic.go:356] "Generic (PLEG): container finished" podID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerID="6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96" exitCode=0
Nov 03 11:22:20.158125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.157891    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"
Nov 03 11:22:20.158125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.157907    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" event={"ID":"fccacc68-1e07-47c4-9878-6bf1dcafbd5b","Type":"ContainerDied","Data":"6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96"}
Nov 03 11:22:20.158125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.158063    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g" event={"ID":"fccacc68-1e07-47c4-9878-6bf1dcafbd5b","Type":"ContainerDied","Data":"3cf5832f806333bbb9ed8b0ba7917f3c34d281e8f4e97729ec2742b67e9f16b8"}
Nov 03 11:22:20.158125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.158118    5393 scope.go:122] "RemoveContainer" containerID="6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96"
Nov 03 11:22:20.168045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.168016    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.168209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.168048    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtxw6\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-kube-api-access-dtxw6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.168209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.168066    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.168209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.168080    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.168209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.168093    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.168209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.168105    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.168209 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.168118    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccacc68-1e07-47c4-9878-6bf1dcafbd5b-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:20.174232 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.174213    5393 scope.go:122] "RemoveContainer" containerID="6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96"
Nov 03 11:22:20.174553 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:20.174527    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96\": container with ID starting with 6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96 not found: ID does not exist" containerID="6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96"
Nov 03 11:22:20.174640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.174565    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96"} err="failed to get container status \"6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96\": rpc error: code = NotFound desc = could not find container \"6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96\": container with ID starting with 6d750bde88f0df6587d5b8609dd32ddbd88340252ef54fcfbcf462d057ce5e96 not found: ID does not exist"
Nov 03 11:22:20.251050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.249038    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"]
Nov 03 11:22:20.251050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.249076    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a68103-openshift-default-b8566486f-p5v4g"]
Nov 03 11:22:20.879662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:20.879627    5393 scope.go:122] "RemoveContainer" containerID="17805146d7a56bbbef61c7748e8442fd3149a6f7e94b931a9666291d110b6f7d"
Nov 03 11:22:21.625480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:21.625431    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:21.640807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:21.637309    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:21.650027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:21.647845    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:22:21.786990 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:21.786954    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:21.788718 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:21.788694    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:21.789866 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:21.789842    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:22:22.235068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:22.235033    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" path="/var/lib/kubelet/pods/fccacc68-1e07-47c4-9878-6bf1dcafbd5b/volumes"
Nov 03 11:22:24.536082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:24.536040    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc" podUID="ed16f6d6-d892-4603-a936-40f5e7ca4e73" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:22:25.192299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:25.192253    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"]
Nov 03 11:22:25.206162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:25.206021    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"]
Nov 03 11:22:28.290802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:28.290715    5393 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","podfeaf2442-f46e-4d47-8b95-6b5465574620"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort podfeaf2442-f46e-4d47-8b95-6b5465574620] : Timed out while waiting for systemd to remove kubepods-besteffort-podfeaf2442_f46e_4d47_8b95_6b5465574620.slice"
Nov 03 11:22:28.290802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:28.290759    5393 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="feaf2442-f46e-4d47-8b95-6b5465574620" err="failed to delete cgroup paths for [kubepods besteffort podfeaf2442-f46e-4d47-8b95-6b5465574620] : unable to destroy cgroup paths for cgroup [kubepods besteffort podfeaf2442-f46e-4d47-8b95-6b5465574620] : Timed out while waiting for systemd to remove kubepods-besteffort-podfeaf2442_f46e_4d47_8b95_6b5465574620.slice"
Nov 03 11:22:29.091059 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:29.091020    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"]
Nov 03 11:22:29.091309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:29.091282    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy" containerID="cri-o://f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a" gracePeriod=30
Nov 03 11:22:30.193631 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.193583    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4" podUID="e3d86bb9-ee25-410c-99d3-adc646178897" containerName="payload-processing" containerID="cri-o://5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496" gracePeriod=25
Nov 03 11:22:30.207160 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.207097    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6" podUID="b8cac8cf-3549-470b-b998-5cf85a1343bc" containerName="payload-pre-processing" containerID="cri-o://d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936" gracePeriod=25
Nov 03 11:22:30.396706 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:22:30.396677    5393 logging.go:55] [core] [Channel #10461 SubChannel #10462]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.23:50052", ServerName: "10.134.1.23:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.23:50052: connect: connection refused"
Nov 03 11:22:30.547665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.547633    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:22:30.597243 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.597216    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:22:30.666050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.651963    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cac8cf-3549-470b-b998-5cf85a1343bc-kube-api-access-vxfcw\" (UniqueName: \"kubernetes.io/projected/b8cac8cf-3549-470b-b998-5cf85a1343bc-kube-api-access-vxfcw\") pod \"b8cac8cf-3549-470b-b998-5cf85a1343bc\" (UID: \"b8cac8cf-3549-470b-b998-5cf85a1343bc\") "
Nov 03 11:22:30.666050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.652442    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cac8cf-3549-470b-b998-5cf85a1343bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cac8cf-3549-470b-b998-5cf85a1343bc-plugins-config-volume\") pod \"b8cac8cf-3549-470b-b998-5cf85a1343bc\" (UID: \"b8cac8cf-3549-470b-b998-5cf85a1343bc\") "
Nov 03 11:22:30.666050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.658139    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cac8cf-3549-470b-b998-5cf85a1343bc-plugins-config-volume" pod "b8cac8cf-3549-470b-b998-5cf85a1343bc" (UID: "b8cac8cf-3549-470b-b998-5cf85a1343bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:30.666050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.661446    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8cac8cf-3549-470b-b998-5cf85a1343bc-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:30.685070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.680982    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cac8cf-3549-470b-b998-5cf85a1343bc-kube-api-access-vxfcw" pod "b8cac8cf-3549-470b-b998-5cf85a1343bc" (UID: "b8cac8cf-3549-470b-b998-5cf85a1343bc"). InnerVolumeSpecName "kube-api-access-vxfcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:30.710759 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.710683    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:22:30.717248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.717216    5393 generic.go:356] "Generic (PLEG): container finished" podID="e3d86bb9-ee25-410c-99d3-adc646178897" containerID="5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496" exitCode=0
Nov 03 11:22:30.717424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.717259    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"
Nov 03 11:22:30.717424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.717293    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4" event={"ID":"e3d86bb9-ee25-410c-99d3-adc646178897","Type":"ContainerDied","Data":"5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496"}
Nov 03 11:22:30.717424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.717348    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4" event={"ID":"e3d86bb9-ee25-410c-99d3-adc646178897","Type":"ContainerDied","Data":"066e8560ff324e239dfeb42c0572d0939481eeafe19d82db75695ca08ec64240"}
Nov 03 11:22:30.717424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.717373    5393 scope.go:122] "RemoveContainer" containerID="5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496"
Nov 03 11:22:30.719376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.719356    5393 generic.go:356] "Generic (PLEG): container finished" podID="ed16f6d6-d892-4603-a936-40f5e7ca4e73" containerID="48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7" exitCode=0
Nov 03 11:22:30.719503 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.719445    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"
Nov 03 11:22:30.719503 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.719480    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc" event={"ID":"ed16f6d6-d892-4603-a936-40f5e7ca4e73","Type":"ContainerDied","Data":"48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7"}
Nov 03 11:22:30.719656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.719524    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc" event={"ID":"ed16f6d6-d892-4603-a936-40f5e7ca4e73","Type":"ContainerDied","Data":"73725ce67a309ed6f828799f3e44468269a97798a2a8cfd886e304cf25b6a6d0"}
Nov 03 11:22:30.721069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.721045    5393 generic.go:356] "Generic (PLEG): container finished" podID="b8cac8cf-3549-470b-b998-5cf85a1343bc" containerID="d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936" exitCode=0
Nov 03 11:22:30.721187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.721086    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"
Nov 03 11:22:30.721187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.721154    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6" event={"ID":"b8cac8cf-3549-470b-b998-5cf85a1343bc","Type":"ContainerDied","Data":"d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936"}
Nov 03 11:22:30.721187 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.721184    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6" event={"ID":"b8cac8cf-3549-470b-b998-5cf85a1343bc","Type":"ContainerDied","Data":"c15f54ccc10ae20598756cce00b6b337996961e985d7f6f9ea3f0a3a5081430b"}
Nov 03 11:22:30.733335 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.733310    5393 scope.go:122] "RemoveContainer" containerID="5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496"
Nov 03 11:22:30.733712 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:30.733687    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496\": container with ID starting with 5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496 not found: ID does not exist" containerID="5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496"
Nov 03 11:22:30.733822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.733719    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496"} err="failed to get container status \"5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496\": rpc error: code = NotFound desc = could not find container \"5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496\": container with ID starting with 5dcb0f093ed71f916915195bcc1e00cbd31d61ee7ec70f6e732a3708ced72496 not found: ID does not exist"
Nov 03 11:22:30.733822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.733750    5393 scope.go:122] "RemoveContainer" containerID="48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7"
Nov 03 11:22:30.745859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.745839    5393 scope.go:122] "RemoveContainer" containerID="48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7"
Nov 03 11:22:30.746234 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:30.746212    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7\": container with ID starting with 48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7 not found: ID does not exist" containerID="48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7"
Nov 03 11:22:30.746332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.746248    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7"} err="failed to get container status \"48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7\": rpc error: code = NotFound desc = could not find container \"48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7\": container with ID starting with 48f42267fbf4d36787736950d7b84ad723859a1fab9f868c6f0d9d90a398bcd7 not found: ID does not exist"
Nov 03 11:22:30.746332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.746271    5393 scope.go:122] "RemoveContainer" containerID="d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936"
Nov 03 11:22:30.756816 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.756786    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"]
Nov 03 11:22:30.761469 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.761429    5393 scope.go:122] "RemoveContainer" containerID="d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936"
Nov 03 11:22:30.762748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.762724    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed16f6d6-d892-4603-a936-40f5e7ca4e73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed16f6d6-d892-4603-a936-40f5e7ca4e73-plugins-config-volume\") pod \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\" (UID: \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\") "
Nov 03 11:22:30.763581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.762772    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed16f6d6-d892-4603-a936-40f5e7ca4e73-kube-api-access-fs4s6\" (UniqueName: \"kubernetes.io/projected/ed16f6d6-d892-4603-a936-40f5e7ca4e73-kube-api-access-fs4s6\") pod \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\" (UID: \"ed16f6d6-d892-4603-a936-40f5e7ca4e73\") "
Nov 03 11:22:30.763581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.763377    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed16f6d6-d892-4603-a936-40f5e7ca4e73-plugins-config-volume" pod "ed16f6d6-d892-4603-a936-40f5e7ca4e73" (UID: "ed16f6d6-d892-4603-a936-40f5e7ca4e73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:30.763886 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:30.763849    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936\": container with ID starting with d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936 not found: ID does not exist" containerID="d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936"
Nov 03 11:22:30.764058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.763891    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936"} err="failed to get container status \"d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936\": rpc error: code = NotFound desc = could not find container \"d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936\": container with ID starting with d72767f47c7166687fc21bf4c702b82b8c3335aa2fe8e07788ae585ad6db9936 not found: ID does not exist"
Nov 03 11:22:30.764261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.764224    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed16f6d6-d892-4603-a936-40f5e7ca4e73-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:30.764261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.764251    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxfcw\" (UniqueName: \"kubernetes.io/projected/b8cac8cf-3549-470b-b998-5cf85a1343bc-kube-api-access-vxfcw\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:30.764656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.764629    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3eb43f-67c99d44bc-zd9c6"]
Nov 03 11:22:30.766903 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.766882    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed16f6d6-d892-4603-a936-40f5e7ca4e73-kube-api-access-fs4s6" pod "ed16f6d6-d892-4603-a936-40f5e7ca4e73" (UID: "ed16f6d6-d892-4603-a936-40f5e7ca4e73"). InnerVolumeSpecName "kube-api-access-fs4s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:30.865047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.865019    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d86bb9-ee25-410c-99d3-adc646178897-kube-api-access-7lxqp\" (UniqueName: \"kubernetes.io/projected/e3d86bb9-ee25-410c-99d3-adc646178897-kube-api-access-7lxqp\") pod \"e3d86bb9-ee25-410c-99d3-adc646178897\" (UID: \"e3d86bb9-ee25-410c-99d3-adc646178897\") "
Nov 03 11:22:30.865216 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.865187    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d86bb9-ee25-410c-99d3-adc646178897-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d86bb9-ee25-410c-99d3-adc646178897-plugins-config-volume\") pod \"e3d86bb9-ee25-410c-99d3-adc646178897\" (UID: \"e3d86bb9-ee25-410c-99d3-adc646178897\") "
Nov 03 11:22:30.865606 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.865582    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d86bb9-ee25-410c-99d3-adc646178897-plugins-config-volume" pod "e3d86bb9-ee25-410c-99d3-adc646178897" (UID: "e3d86bb9-ee25-410c-99d3-adc646178897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:30.866045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.866025    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs4s6\" (UniqueName: \"kubernetes.io/projected/ed16f6d6-d892-4603-a936-40f5e7ca4e73-kube-api-access-fs4s6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:30.866097 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.866049    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3d86bb9-ee25-410c-99d3-adc646178897-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:30.867249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.867224    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d86bb9-ee25-410c-99d3-adc646178897-kube-api-access-7lxqp" pod "e3d86bb9-ee25-410c-99d3-adc646178897" (UID: "e3d86bb9-ee25-410c-99d3-adc646178897"). InnerVolumeSpecName "kube-api-access-7lxqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:30.967662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:30.967577    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lxqp\" (UniqueName: \"kubernetes.io/projected/e3d86bb9-ee25-410c-99d3-adc646178897-kube-api-access-7lxqp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:31.045346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:31.045315    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"]
Nov 03 11:22:31.050870 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:31.050843    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4"]
Nov 03 11:22:31.060951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:31.058347    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"]
Nov 03 11:22:31.062505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:31.062481    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a68103-54dc494f95-66nnc"]
Nov 03 11:22:31.395010 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:31.394965    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-3eb43f-7fbc776f89-p5hh4" podUID="e3d86bb9-ee25-410c-99d3-adc646178897" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.23:50052\" within 1s: context deadline exceeded"
Nov 03 11:22:31.823425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:31.823336    5393 patch_prober.go:29] interesting pod/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:22:31.823425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:31.823392    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:22:32.256445 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:32.256404    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cac8cf-3549-470b-b998-5cf85a1343bc" path="/var/lib/kubelet/pods/b8cac8cf-3549-470b-b998-5cf85a1343bc/volumes"
Nov 03 11:22:32.257164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:32.257139    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d86bb9-ee25-410c-99d3-adc646178897" path="/var/lib/kubelet/pods/e3d86bb9-ee25-410c-99d3-adc646178897/volumes"
Nov 03 11:22:32.257811 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:32.257723    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed16f6d6-d892-4603-a936-40f5e7ca4e73" path="/var/lib/kubelet/pods/ed16f6d6-d892-4603-a936-40f5e7ca4e73/volumes"
Nov 03 11:22:34.452266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.452240    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:22:34.549386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.549348    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-data\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.549748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.549419    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-certs\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.549748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.549664    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-data" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.549748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.549728    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-certs" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.550049 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.550019    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-envoy\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.550182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.550063    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istiod-ca-cert\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.550182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.550113    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-podinfo\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.550182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.550153    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-kube-api-access-hmn7c\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-kube-api-access-hmn7c\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.550379 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.550209    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-credential-socket\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.550379 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.550282    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-token\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-token\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.550495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.550390    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-socket\") pod \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\" (UID: \"ace83d08-f3da-4b9d-abfd-e8050fa828a5\") "
Nov 03 11:22:34.550623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.550598    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istiod-ca-cert" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:22:34.551134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.551107    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-credential-socket" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.551374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.551349    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-socket" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.551615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.551584    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.551758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.551742    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.551871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.551857    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.552001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.551986    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.553822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.553794    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-podinfo" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:22:34.554504 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.554477    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-token" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:34.554632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.554612    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-envoy" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:34.557325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.557277    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-kube-api-access-hmn7c" pod "ace83d08-f3da-4b9d-abfd-e8050fa828a5" (UID: "ace83d08-f3da-4b9d-abfd-e8050fa828a5"). InnerVolumeSpecName "kube-api-access-hmn7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:34.653712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.653675    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.653712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.653706    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.653712 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.653717    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmn7c\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-kube-api-access-hmn7c\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.653988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.653726    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace83d08-f3da-4b9d-abfd-e8050fa828a5-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.653988 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.653735    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace83d08-f3da-4b9d-abfd-e8050fa828a5-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:34.760711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.760619    5393 generic.go:356] "Generic (PLEG): container finished" podID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerID="f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a" exitCode=0
Nov 03 11:22:34.760711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.760671    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"
Nov 03 11:22:34.760711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.760695    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" event={"ID":"ace83d08-f3da-4b9d-abfd-e8050fa828a5","Type":"ContainerDied","Data":"f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a"}
Nov 03 11:22:34.761029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.760749    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4" event={"ID":"ace83d08-f3da-4b9d-abfd-e8050fa828a5","Type":"ContainerDied","Data":"4343cf6f31db35590d079bf8c2b9815e9f199e1f75b6cb6e79e1dfa25b43453d"}
Nov 03 11:22:34.761029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.760771    5393 scope.go:122] "RemoveContainer" containerID="f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a"
Nov 03 11:22:34.773174 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.773156    5393 scope.go:122] "RemoveContainer" containerID="1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f"
Nov 03 11:22:34.784307 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.784278    5393 scope.go:122] "RemoveContainer" containerID="f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a"
Nov 03 11:22:34.784643 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:34.784614    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a\": container with ID starting with f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a not found: ID does not exist" containerID="f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a"
Nov 03 11:22:34.784693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.784657    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a"} err="failed to get container status \"f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a\": rpc error: code = NotFound desc = could not find container \"f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a\": container with ID starting with f5df218340f0a66063f176edf395fb1bb40f9dcdd9ffc4595797073a8033127a not found: ID does not exist"
Nov 03 11:22:34.784693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.784683    5393 scope.go:122] "RemoveContainer" containerID="1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f"
Nov 03 11:22:34.784981 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:34.784958    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f\": container with ID starting with 1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f not found: ID does not exist" containerID="1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f"
Nov 03 11:22:34.785073 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.784987    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f"} err="failed to get container status \"1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f\": rpc error: code = NotFound desc = could not find container \"1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f\": container with ID starting with 1456416731e886f63ae723ee3e2a202be41dd2b28c624720c2551ad9a25c8c2f not found: ID does not exist"
Nov 03 11:22:34.791120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.791090    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"]
Nov 03 11:22:34.796625 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:34.796604    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24376f-openshift-default-8df4546fc-85hw4"]
Nov 03 11:22:36.234299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:36.234268    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" path="/var/lib/kubelet/pods/ace83d08-f3da-4b9d-abfd-e8050fa828a5/volumes"
Nov 03 11:22:43.261021 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:43.260980    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:22:43.771423 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:43.771381    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"]
Nov 03 11:22:54.897941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:54.889376    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"]
Nov 03 11:22:57.929440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:57.929402    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"]
Nov 03 11:22:57.929863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:57.929617    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy" containerID="cri-o://4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412" gracePeriod=30
Nov 03 11:22:58.795515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:58.795475    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" containerName="main" containerID="cri-o://93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3" gracePeriod=45
Nov 03 11:22:58.800877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:58.800836    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" containerName="main" probeResult="failure" output="Get \"https://10.134.1.37:8000/ready\": EOF"
Nov 03 11:22:59.099796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.099764    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:22:59.141864 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.141811    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-home\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-home\") pod \"a6934784-f140-4ebb-af7d-b418d1238e93\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") "
Nov 03 11:22:59.142081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.141905    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6934784-f140-4ebb-af7d-b418d1238e93-kube-api-access-6p4lm\" (UniqueName: \"kubernetes.io/projected/a6934784-f140-4ebb-af7d-b418d1238e93-kube-api-access-6p4lm\") pod \"a6934784-f140-4ebb-af7d-b418d1238e93\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") "
Nov 03 11:22:59.142081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.141993    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6934784-f140-4ebb-af7d-b418d1238e93-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6934784-f140-4ebb-af7d-b418d1238e93-tls-certs\") pod \"a6934784-f140-4ebb-af7d-b418d1238e93\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") "
Nov 03 11:22:59.142222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.142119    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-home" pod "a6934784-f140-4ebb-af7d-b418d1238e93" (UID: "a6934784-f140-4ebb-af7d-b418d1238e93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.142289 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.142266    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-tmp-dir\") pod \"a6934784-f140-4ebb-af7d-b418d1238e93\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") "
Nov 03 11:22:59.142343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.142303    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-model-cache\") pod \"a6934784-f140-4ebb-af7d-b418d1238e93\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") "
Nov 03 11:22:59.142380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.142339    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-dshm\") pod \"a6934784-f140-4ebb-af7d-b418d1238e93\" (UID: \"a6934784-f140-4ebb-af7d-b418d1238e93\") "
Nov 03 11:22:59.143770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.142825    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-tmp-dir" pod "a6934784-f140-4ebb-af7d-b418d1238e93" (UID: "a6934784-f140-4ebb-af7d-b418d1238e93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.143770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.143102    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.143770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.143124    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.143770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.143374    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-model-cache" pod "a6934784-f140-4ebb-af7d-b418d1238e93" (UID: "a6934784-f140-4ebb-af7d-b418d1238e93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.145145 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.145112    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6934784-f140-4ebb-af7d-b418d1238e93-kube-api-access-6p4lm" pod "a6934784-f140-4ebb-af7d-b418d1238e93" (UID: "a6934784-f140-4ebb-af7d-b418d1238e93"). InnerVolumeSpecName "kube-api-access-6p4lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:22:59.145366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.145343    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-dshm" pod "a6934784-f140-4ebb-af7d-b418d1238e93" (UID: "a6934784-f140-4ebb-af7d-b418d1238e93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:22:59.146078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.146053    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6934784-f140-4ebb-af7d-b418d1238e93-tls-certs" pod "a6934784-f140-4ebb-af7d-b418d1238e93" (UID: "a6934784-f140-4ebb-af7d-b418d1238e93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:22:59.258977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.257085    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.258977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.257125    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6934784-f140-4ebb-af7d-b418d1238e93-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.258977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.257143    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4lm\" (UniqueName: \"kubernetes.io/projected/a6934784-f140-4ebb-af7d-b418d1238e93-kube-api-access-6p4lm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.258977 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.257160    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6934784-f140-4ebb-af7d-b418d1238e93-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:22:59.304851 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.304814    5393 generic.go:356] "Generic (PLEG): container finished" podID="a6934784-f140-4ebb-af7d-b418d1238e93" containerID="93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3" exitCode=0
Nov 03 11:22:59.305084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.304875    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"
Nov 03 11:22:59.305084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.304895    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" event={"ID":"a6934784-f140-4ebb-af7d-b418d1238e93","Type":"ContainerDied","Data":"93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3"}
Nov 03 11:22:59.305084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.304986    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr" event={"ID":"a6934784-f140-4ebb-af7d-b418d1238e93","Type":"ContainerDied","Data":"39837ae9d0e4bb94afdc78385c76be75ccdb71ba92aa5756ed586a93d51b041d"}
Nov 03 11:22:59.305084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.305010    5393 scope.go:122] "RemoveContainer" containerID="93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3"
Nov 03 11:22:59.317501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.317481    5393 scope.go:122] "RemoveContainer" containerID="93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3"
Nov 03 11:22:59.317821 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:22:59.317800    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3\": container with ID starting with 93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3 not found: ID does not exist" containerID="93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3"
Nov 03 11:22:59.317876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.317832    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3"} err="failed to get container status \"93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3\": rpc error: code = NotFound desc = could not find container \"93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3\": container with ID starting with 93b36ea28c0b47d17d12c2ce2bf4c256ab93f183dd6c89a5748e8cd5c33dc3b3 not found: ID does not exist"
Nov 03 11:22:59.329595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.329521    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"]
Nov 03 11:22:59.333472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:22:59.333445    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-68aa46/test-model-68aa46-kserve-6b7bcc9d47-dtmsr"]
Nov 03 11:23:00.233095 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:00.233062    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" path="/var/lib/kubelet/pods/a6934784-f140-4ebb-af7d-b418d1238e93/volumes"
Nov 03 11:23:03.224902 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.224876    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:23:03.330373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330333    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-credential-socket\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.330575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330411    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-certs\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.330575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330503    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-data\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.330700 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330602    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7266464-203d-484f-8c3c-265e27639945-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7266464-203d-484f-8c3c-265e27639945-istiod-ca-cert\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.330700 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330627    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-credential-socket" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:03.330700 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330694    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-kube-api-access-df9fv\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-kube-api-access-df9fv\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.330871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330735    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-certs" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:03.330871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330751    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-data" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:03.330871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330865    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-istio-token\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-istio-token\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.331078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330935    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7266464-203d-484f-8c3c-265e27639945-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7266464-203d-484f-8c3c-265e27639945-istio-podinfo\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.331078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330968    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7266464-203d-484f-8c3c-265e27639945-istiod-ca-cert" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:03.331078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.330982    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-envoy\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.331078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.331073    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-socket\") pod \"f7266464-203d-484f-8c3c-265e27639945\" (UID: \"f7266464-203d-484f-8c3c-265e27639945\") "
Nov 03 11:23:03.331390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.331365    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-socket" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:03.331829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.331810    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7266464-203d-484f-8c3c-265e27639945-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:03.331958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.331834    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:03.331958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.331851    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:03.331958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.331877    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:03.331958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.331887    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:03.333441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.333414    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-kube-api-access-df9fv" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "kube-api-access-df9fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:03.333579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.333564    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7266464-203d-484f-8c3c-265e27639945-istio-podinfo" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:23:03.333647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.333616    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-istio-token" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:03.334199 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.334164    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-envoy" pod "f7266464-203d-484f-8c3c-265e27639945" (UID: "f7266464-203d-484f-8c3c-265e27639945"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:03.343092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.343064    5393 generic.go:356] "Generic (PLEG): container finished" podID="f7266464-203d-484f-8c3c-265e27639945" containerID="4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412" exitCode=0
Nov 03 11:23:03.343223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.343119    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"
Nov 03 11:23:03.343223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.343150    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" event={"ID":"f7266464-203d-484f-8c3c-265e27639945","Type":"ContainerDied","Data":"4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412"}
Nov 03 11:23:03.343331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.343236    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6" event={"ID":"f7266464-203d-484f-8c3c-265e27639945","Type":"ContainerDied","Data":"2f6a3c46bf4489648284ca0c76623571bf0892baeda1491b9e85b9922c5a309f"}
Nov 03 11:23:03.343331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.343276    5393 scope.go:122] "RemoveContainer" containerID="4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412"
Nov 03 11:23:03.376296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.376273    5393 scope.go:122] "RemoveContainer" containerID="4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412"
Nov 03 11:23:03.376675 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:23:03.376653    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412\": container with ID starting with 4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412 not found: ID does not exist" containerID="4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412"
Nov 03 11:23:03.376763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.376688    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412"} err="failed to get container status \"4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412\": rpc error: code = NotFound desc = could not find container \"4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412\": container with ID starting with 4f90ec03157b882a106ddcf87c8ac84af795c4b6641b96a0054881f017ff8412 not found: ID does not exist"
Nov 03 11:23:03.388547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.388519    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"]
Nov 03 11:23:03.393479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.393444    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fb5766-openshift-default-79bb9d756f-9lct6"]
Nov 03 11:23:03.433975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.433942    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9fv\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-kube-api-access-df9fv\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:03.433975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.433971    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7266464-203d-484f-8c3c-265e27639945-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:03.433975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.433983    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7266464-203d-484f-8c3c-265e27639945-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:03.434235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:03.433993    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7266464-203d-484f-8c3c-265e27639945-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:04.233294 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:04.233262    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7266464-203d-484f-8c3c-265e27639945" path="/var/lib/kubelet/pods/f7266464-203d-484f-8c3c-265e27639945/volumes"
Nov 03 11:23:05.386575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:05.386539    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"]
Nov 03 11:23:05.405860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:05.405829    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"]
Nov 03 11:23:09.925059 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:09.924981    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg" podUID="f00dd0cb-ef27-428f-b926-21396cb38b43" containerName="main" containerID="cri-o://31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52" gracePeriod=45
Nov 03 11:23:10.249000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.242415    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.463067    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9" podUID="9fffc3d5-3d17-471e-8906-5a94ed21d14c" containerName="payload-pre-processing" containerID="cri-o://d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269" gracePeriod=25
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.463250    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k" podUID="c6583710-32ad-4a57-9935-cb8953eaea21" containerName="payload-processing" containerID="cri-o://40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae" gracePeriod=25
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.464291    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00dd0cb-ef27-428f-b926-21396cb38b43-kube-api-access-2nvjd\" (UniqueName: \"kubernetes.io/projected/f00dd0cb-ef27-428f-b926-21396cb38b43-kube-api-access-2nvjd\") pod \"f00dd0cb-ef27-428f-b926-21396cb38b43\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") "
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.464354    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-home\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-home\") pod \"f00dd0cb-ef27-428f-b926-21396cb38b43\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") "
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.464448    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-model-cache\") pod \"f00dd0cb-ef27-428f-b926-21396cb38b43\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") "
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.464609    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f00dd0cb-ef27-428f-b926-21396cb38b43-tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dd0cb-ef27-428f-b926-21396cb38b43-tls-certs\") pod \"f00dd0cb-ef27-428f-b926-21396cb38b43\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") "
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.464772    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-tmp-dir\") pod \"f00dd0cb-ef27-428f-b926-21396cb38b43\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") "
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.464804    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-dshm\") pod \"f00dd0cb-ef27-428f-b926-21396cb38b43\" (UID: \"f00dd0cb-ef27-428f-b926-21396cb38b43\") "
Nov 03 11:23:10.466128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.465204    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-model-cache" pod "f00dd0cb-ef27-428f-b926-21396cb38b43" (UID: "f00dd0cb-ef27-428f-b926-21396cb38b43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:10.484978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.469192    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-home" pod "f00dd0cb-ef27-428f-b926-21396cb38b43" (UID: "f00dd0cb-ef27-428f-b926-21396cb38b43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:10.484978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.470361    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-tmp-dir" pod "f00dd0cb-ef27-428f-b926-21396cb38b43" (UID: "f00dd0cb-ef27-428f-b926-21396cb38b43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:10.484978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.471478    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:10.484978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.471501    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:10.484978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.471516    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:10.484978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.471974    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-dshm" pod "f00dd0cb-ef27-428f-b926-21396cb38b43" (UID: "f00dd0cb-ef27-428f-b926-21396cb38b43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:10.484978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.479061    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00dd0cb-ef27-428f-b926-21396cb38b43-tls-certs" pod "f00dd0cb-ef27-428f-b926-21396cb38b43" (UID: "f00dd0cb-ef27-428f-b926-21396cb38b43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:23:10.484978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.479103    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00dd0cb-ef27-428f-b926-21396cb38b43-kube-api-access-2nvjd" pod "f00dd0cb-ef27-428f-b926-21396cb38b43" (UID: "f00dd0cb-ef27-428f-b926-21396cb38b43"). InnerVolumeSpecName "kube-api-access-2nvjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:10.501211 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.501176    5393 generic.go:356] "Generic (PLEG): container finished" podID="f00dd0cb-ef27-428f-b926-21396cb38b43" containerID="31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52" exitCode=0
Nov 03 11:23:10.501383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.501234    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg" event={"ID":"f00dd0cb-ef27-428f-b926-21396cb38b43","Type":"ContainerDied","Data":"31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52"}
Nov 03 11:23:10.501383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.501262    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg" event={"ID":"f00dd0cb-ef27-428f-b926-21396cb38b43","Type":"ContainerDied","Data":"c69055d54b569949b51f1a2439de7ca5ffa9f65253d2ee947736f85144667993"}
Nov 03 11:23:10.501383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.501282    5393 scope.go:122] "RemoveContainer" containerID="31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52"
Nov 03 11:23:10.501601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.501408    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"
Nov 03 11:23:10.564997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.564964    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"]
Nov 03 11:23:10.568755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.568734    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c0a748/test-model-c0a748-kserve-78d66bfc69-vz7wg"]
Nov 03 11:23:10.573145 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.573120    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dd0cb-ef27-428f-b926-21396cb38b43-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:10.573145 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.573143    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dd0cb-ef27-428f-b926-21396cb38b43-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:10.573319 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.573152    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nvjd\" (UniqueName: \"kubernetes.io/projected/f00dd0cb-ef27-428f-b926-21396cb38b43-kube-api-access-2nvjd\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:10.602225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.602205    5393 scope.go:122] "RemoveContainer" containerID="31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52"
Nov 03 11:23:10.602610 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:23:10.602584    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52\": container with ID starting with 31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52 not found: ID does not exist" containerID="31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52"
Nov 03 11:23:10.602718 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.602620    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52"} err="failed to get container status \"31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52\": rpc error: code = NotFound desc = could not find container \"31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52\": container with ID starting with 31ab74f56f7235b4585ec433f27553a680b3bf85fd6374b3d57f83df5e3cec52 not found: ID does not exist"
Nov 03 11:23:10.760265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.760236    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:23:10.877794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.877759    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6583710-32ad-4a57-9935-cb8953eaea21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6583710-32ad-4a57-9935-cb8953eaea21-plugins-config-volume\") pod \"c6583710-32ad-4a57-9935-cb8953eaea21\" (UID: \"c6583710-32ad-4a57-9935-cb8953eaea21\") "
Nov 03 11:23:10.877987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.877954    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6583710-32ad-4a57-9935-cb8953eaea21-kube-api-access-wbbmt\" (UniqueName: \"kubernetes.io/projected/c6583710-32ad-4a57-9935-cb8953eaea21-kube-api-access-wbbmt\") pod \"c6583710-32ad-4a57-9935-cb8953eaea21\" (UID: \"c6583710-32ad-4a57-9935-cb8953eaea21\") "
Nov 03 11:23:10.878147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.878126    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6583710-32ad-4a57-9935-cb8953eaea21-plugins-config-volume" pod "c6583710-32ad-4a57-9935-cb8953eaea21" (UID: "c6583710-32ad-4a57-9935-cb8953eaea21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:10.878488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.878473    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6583710-32ad-4a57-9935-cb8953eaea21-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:10.880195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.880170    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6583710-32ad-4a57-9935-cb8953eaea21-kube-api-access-wbbmt" pod "c6583710-32ad-4a57-9935-cb8953eaea21" (UID: "c6583710-32ad-4a57-9935-cb8953eaea21"). InnerVolumeSpecName "kube-api-access-wbbmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:10.979807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:10.979764    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbbmt\" (UniqueName: \"kubernetes.io/projected/c6583710-32ad-4a57-9935-cb8953eaea21-kube-api-access-wbbmt\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:11.513460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.513429    5393 generic.go:356] "Generic (PLEG): container finished" podID="c6583710-32ad-4a57-9935-cb8953eaea21" containerID="40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae" exitCode=0
Nov 03 11:23:11.513694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.513478    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"
Nov 03 11:23:11.513694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.513516    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k" event={"ID":"c6583710-32ad-4a57-9935-cb8953eaea21","Type":"ContainerDied","Data":"40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae"}
Nov 03 11:23:11.513694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.513576    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k" event={"ID":"c6583710-32ad-4a57-9935-cb8953eaea21","Type":"ContainerDied","Data":"150add15c9e5a3df5de34865dd6e74389c5ac620656d3b7819b534a15e526306"}
Nov 03 11:23:11.513694 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.513606    5393 scope.go:122] "RemoveContainer" containerID="40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae"
Nov 03 11:23:11.525704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.525683    5393 scope.go:122] "RemoveContainer" containerID="40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae"
Nov 03 11:23:11.526031 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:23:11.526005    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae\": container with ID starting with 40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae not found: ID does not exist" containerID="40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae"
Nov 03 11:23:11.526138 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.526036    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae"} err="failed to get container status \"40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae\": rpc error: code = NotFound desc = could not find container \"40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae\": container with ID starting with 40b4beba2051f4b129a4a15abc7ee678551f3dcd1405062b4d7704152453a3ae not found: ID does not exist"
Nov 03 11:23:11.537715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.537689    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"]
Nov 03 11:23:11.543879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:11.543855    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-68aa46-85cbd56d69-8fh6k"]
Nov 03 11:23:12.233807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:12.233776    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6583710-32ad-4a57-9935-cb8953eaea21" path="/var/lib/kubelet/pods/c6583710-32ad-4a57-9935-cb8953eaea21/volumes"
Nov 03 11:23:12.234613 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:12.234583    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00dd0cb-ef27-428f-b926-21396cb38b43" path="/var/lib/kubelet/pods/f00dd0cb-ef27-428f-b926-21396cb38b43/volumes"
Nov 03 11:23:13.196664 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:23:13.196627    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:23:13.881777 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:13.881733    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"]
Nov 03 11:23:13.882297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:13.882008    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy" containerID="cri-o://10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc" gracePeriod=30
Nov 03 11:23:19.179941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.179895    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:23:19.278004 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.277968    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-token\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-token\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278045    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-envoy\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278128    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8713c92b-ecbc-4b24-82b7-055b9dc84386-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8713c92b-ecbc-4b24-82b7-055b9dc84386-istiod-ca-cert\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278352 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278238    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-data\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278352 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278255    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-podinfo\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278352 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278277    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-socket\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278422    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-credential-socket\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278515    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-certs\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278566    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-kube-api-access-t7zpb\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-kube-api-access-t7zpb\") pod \"8713c92b-ecbc-4b24-82b7-055b9dc84386\" (UID: \"8713c92b-ecbc-4b24-82b7-055b9dc84386\") "
Nov 03 11:23:19.278749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278580    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-data" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.278749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278674    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8713c92b-ecbc-4b24-82b7-055b9dc84386-istiod-ca-cert" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:19.278749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278668    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-socket" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.279103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278840    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-credential-socket" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.279103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.278858    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-certs" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.279261 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.279240    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.279325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.279263    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.279325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.279279    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8713c92b-ecbc-4b24-82b7-055b9dc84386-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.279325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.279289    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.279325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.279298    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.280806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.280775    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-envoy" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:19.281009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.280876    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-podinfo" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:23:19.281188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.281172    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-kube-api-access-t7zpb" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "kube-api-access-t7zpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:19.281398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.281376    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-token" pod "8713c92b-ecbc-4b24-82b7-055b9dc84386" (UID: "8713c92b-ecbc-4b24-82b7-055b9dc84386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:19.382641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.382130    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zpb\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-kube-api-access-t7zpb\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.382641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.382257    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.382641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.382275    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.382641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.382288    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8713c92b-ecbc-4b24-82b7-055b9dc84386-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:19.404945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.402410    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9" podUID="9fffc3d5-3d17-471e-8906-5a94ed21d14c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:23:19.604195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.604164    5393 generic.go:356] "Generic (PLEG): container finished" podID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerID="10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc" exitCode=0
Nov 03 11:23:19.604402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.604213    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"
Nov 03 11:23:19.604402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.604248    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" event={"ID":"8713c92b-ecbc-4b24-82b7-055b9dc84386","Type":"ContainerDied","Data":"10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc"}
Nov 03 11:23:19.604402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.604306    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2" event={"ID":"8713c92b-ecbc-4b24-82b7-055b9dc84386","Type":"ContainerDied","Data":"8e9dee19d54bef4c60b41f81b4af9ce2fbae44145128483d59355679e7780f9a"}
Nov 03 11:23:19.604402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.604336    5393 scope.go:122] "RemoveContainer" containerID="10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc"
Nov 03 11:23:19.617898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.617877    5393 scope.go:122] "RemoveContainer" containerID="10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc"
Nov 03 11:23:19.618333 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:23:19.618189    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc\": container with ID starting with 10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc not found: ID does not exist" containerID="10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc"
Nov 03 11:23:19.618333 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.618217    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc"} err="failed to get container status \"10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc\": rpc error: code = NotFound desc = could not find container \"10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc\": container with ID starting with 10755b67586cb41fe8e59cd381bc9bc1f4434e438df1aa6328c3084e261256bc not found: ID does not exist"
Nov 03 11:23:19.629805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.629772    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"]
Nov 03 11:23:19.633636 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:19.633611    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9f53bb-openshift-default-567b58d445-4bqf2"]
Nov 03 11:23:20.232266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:20.232233    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" path="/var/lib/kubelet/pods/8713c92b-ecbc-4b24-82b7-055b9dc84386/volumes"
Nov 03 11:23:29.400125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:29.400086    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9" podUID="9fffc3d5-3d17-471e-8906-5a94ed21d14c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:23:30.725377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.725344    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:23:30.735341 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.735310    5393 generic.go:356] "Generic (PLEG): container finished" podID="9fffc3d5-3d17-471e-8906-5a94ed21d14c" containerID="d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269" exitCode=0
Nov 03 11:23:30.735499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.735361    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"
Nov 03 11:23:30.735499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.735391    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9" event={"ID":"9fffc3d5-3d17-471e-8906-5a94ed21d14c","Type":"ContainerDied","Data":"d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269"}
Nov 03 11:23:30.735499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.735454    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9" event={"ID":"9fffc3d5-3d17-471e-8906-5a94ed21d14c","Type":"ContainerDied","Data":"e9e53af48ecfb1c43978fa81741c008d8921c535894e62ac9d1245edbafeed84"}
Nov 03 11:23:30.735499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.735479    5393 scope.go:122] "RemoveContainer" containerID="d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269"
Nov 03 11:23:30.747992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.747895    5393 scope.go:122] "RemoveContainer" containerID="d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269"
Nov 03 11:23:30.748334 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:23:30.748299    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269\": container with ID starting with d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269 not found: ID does not exist" containerID="d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269"
Nov 03 11:23:30.748419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.748347    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269"} err="failed to get container status \"d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269\": rpc error: code = NotFound desc = could not find container \"d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269\": container with ID starting with d0037417e42de718feef38d1bbebdb30b56ce1ba97398da3f17c19f138478269 not found: ID does not exist"
Nov 03 11:23:30.835576 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.835542    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fffc3d5-3d17-471e-8906-5a94ed21d14c-kube-api-access-xcrxp\" (UniqueName: \"kubernetes.io/projected/9fffc3d5-3d17-471e-8906-5a94ed21d14c-kube-api-access-xcrxp\") pod \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\" (UID: \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\") "
Nov 03 11:23:30.835810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.835795    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fffc3d5-3d17-471e-8906-5a94ed21d14c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fffc3d5-3d17-471e-8906-5a94ed21d14c-plugins-config-volume\") pod \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\" (UID: \"9fffc3d5-3d17-471e-8906-5a94ed21d14c\") "
Nov 03 11:23:30.836256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.836229    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fffc3d5-3d17-471e-8906-5a94ed21d14c-plugins-config-volume" pod "9fffc3d5-3d17-471e-8906-5a94ed21d14c" (UID: "9fffc3d5-3d17-471e-8906-5a94ed21d14c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:30.836543 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.836528    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fffc3d5-3d17-471e-8906-5a94ed21d14c-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:30.837975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.837939    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fffc3d5-3d17-471e-8906-5a94ed21d14c-kube-api-access-xcrxp" pod "9fffc3d5-3d17-471e-8906-5a94ed21d14c" (UID: "9fffc3d5-3d17-471e-8906-5a94ed21d14c"). InnerVolumeSpecName "kube-api-access-xcrxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:30.937820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:30.937784    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcrxp\" (UniqueName: \"kubernetes.io/projected/9fffc3d5-3d17-471e-8906-5a94ed21d14c-kube-api-access-xcrxp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:31.060186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:31.060154    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"]
Nov 03 11:23:31.064345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:31.064313    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68aa46-7fbb47ffc9-2zvd9"]
Nov 03 11:23:32.025891 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:32.025861    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"]
Nov 03 11:23:32.026741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:32.026132    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5" podUID="5e2befaa-bf7f-4806-9400-f40c87ab741c" containerName="maas-api" containerID="cri-o://9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d" gracePeriod=30
Nov 03 11:23:32.237729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:32.237686    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fffc3d5-3d17-471e-8906-5a94ed21d14c" path="/var/lib/kubelet/pods/9fffc3d5-3d17-471e-8906-5a94ed21d14c/volumes"
Nov 03 11:23:33.333689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.333659    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:23:33.396755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.396663    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-metrics-certs\") pod \"5e2befaa-bf7f-4806-9400-f40c87ab741c\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") "
Nov 03 11:23:33.396960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.396767    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e2befaa-bf7f-4806-9400-f40c87ab741c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2befaa-bf7f-4806-9400-f40c87ab741c-openshift-service-ca\") pod \"5e2befaa-bf7f-4806-9400-f40c87ab741c\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") "
Nov 03 11:23:33.397170 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.397145    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-maas-api-tls\") pod \"5e2befaa-bf7f-4806-9400-f40c87ab741c\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") "
Nov 03 11:23:33.397287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.397195    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2befaa-bf7f-4806-9400-f40c87ab741c-openshift-service-ca" pod "5e2befaa-bf7f-4806-9400-f40c87ab741c" (UID: "5e2befaa-bf7f-4806-9400-f40c87ab741c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:33.397357 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.397281    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2befaa-bf7f-4806-9400-f40c87ab741c-kube-api-access-hvdc2\" (UniqueName: \"kubernetes.io/projected/5e2befaa-bf7f-4806-9400-f40c87ab741c-kube-api-access-hvdc2\") pod \"5e2befaa-bf7f-4806-9400-f40c87ab741c\" (UID: \"5e2befaa-bf7f-4806-9400-f40c87ab741c\") "
Nov 03 11:23:33.398282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.398256    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e2befaa-bf7f-4806-9400-f40c87ab741c-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:33.399847 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.399814    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-maas-api-tls" pod "5e2befaa-bf7f-4806-9400-f40c87ab741c" (UID: "5e2befaa-bf7f-4806-9400-f40c87ab741c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:23:33.400195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.400160    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-metrics-certs" pod "5e2befaa-bf7f-4806-9400-f40c87ab741c" (UID: "5e2befaa-bf7f-4806-9400-f40c87ab741c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:23:33.400297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.400270    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2befaa-bf7f-4806-9400-f40c87ab741c-kube-api-access-hvdc2" pod "5e2befaa-bf7f-4806-9400-f40c87ab741c" (UID: "5e2befaa-bf7f-4806-9400-f40c87ab741c"). InnerVolumeSpecName "kube-api-access-hvdc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:33.508173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.508076    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvdc2\" (UniqueName: \"kubernetes.io/projected/5e2befaa-bf7f-4806-9400-f40c87ab741c-kube-api-access-hvdc2\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:33.508173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.508123    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:33.508173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.508142    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e2befaa-bf7f-4806-9400-f40c87ab741c-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:33.760255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.760159    5393 generic.go:356] "Generic (PLEG): container finished" podID="5e2befaa-bf7f-4806-9400-f40c87ab741c" containerID="9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d" exitCode=0
Nov 03 11:23:33.760255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.760219    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"
Nov 03 11:23:33.760460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.760253    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5" event={"ID":"5e2befaa-bf7f-4806-9400-f40c87ab741c","Type":"ContainerDied","Data":"9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d"}
Nov 03 11:23:33.760460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.760300    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5" event={"ID":"5e2befaa-bf7f-4806-9400-f40c87ab741c","Type":"ContainerDied","Data":"52f1ec6102c90f1e2926a8d7684e0412bde447f18be558b3642a1635161445d0"}
Nov 03 11:23:33.760460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.760324    5393 scope.go:122] "RemoveContainer" containerID="9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d"
Nov 03 11:23:33.772693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.772673    5393 scope.go:122] "RemoveContainer" containerID="9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d"
Nov 03 11:23:33.773048 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:23:33.773023    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d\": container with ID starting with 9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d not found: ID does not exist" containerID="9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d"
Nov 03 11:23:33.773148 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.773063    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d"} err="failed to get container status \"9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d\": rpc error: code = NotFound desc = could not find container \"9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d\": container with ID starting with 9adf4ed7c416290169e7955f7d20d1323bc2224a8f80dcb034cf61824e065b2d not found: ID does not exist"
Nov 03 11:23:33.785597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.785570    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"]
Nov 03 11:23:33.789550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:33.789526    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c0a748-85c96c556d-7ntl5"]
Nov 03 11:23:34.232791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:34.232759    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e2befaa-bf7f-4806-9400-f40c87ab741c" path="/var/lib/kubelet/pods/5e2befaa-bf7f-4806-9400-f40c87ab741c/volumes"
Nov 03 11:23:39.568506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:39.568471    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"]
Nov 03 11:23:39.569335 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:39.569304    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy" containerID="cri-o://faf88e517aa6c7af871e0cf7ace0817e08152da08b83df45b0133c8a9618fa3f" gracePeriod=30
Nov 03 11:23:39.574787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:39.574760    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"]
Nov 03 11:23:39.575064 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:39.575032    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy" containerID="cri-o://0f4f0e8250b3dbabad9cbc47857acf47799aafe5e308bf06e54479f36d933be4" gracePeriod=30
Nov 03 11:23:41.141231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.141180    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"]
Nov 03 11:23:41.142383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142361    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e3d86bb9-ee25-410c-99d3-adc646178897" containerName="payload-processing"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142385    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d86bb9-ee25-410c-99d3-adc646178897" containerName="payload-processing"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142427    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c6583710-32ad-4a57-9935-cb8953eaea21" containerName="payload-processing"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142435    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6583710-32ad-4a57-9935-cb8953eaea21" containerName="payload-processing"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142454    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5e2befaa-bf7f-4806-9400-f40c87ab741c" containerName="maas-api"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142462    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2befaa-bf7f-4806-9400-f40c87ab741c" containerName="maas-api"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142477    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142485    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142517    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy"
Nov 03 11:23:41.142522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142524    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142547    5393 cpu_manager.go:414] "containerMap: removing container" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142554    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142580    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142587    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142603    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b8cac8cf-3549-470b-b998-5cf85a1343bc" containerName="payload-pre-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142610    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cac8cf-3549-470b-b998-5cf85a1343bc" containerName="payload-pre-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142620    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" containerName="main"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142627    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" containerName="main"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142647    5393 cpu_manager.go:414] "containerMap: removing container" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142655    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142669    5393 cpu_manager.go:414] "containerMap: removing container" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142676    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142693    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f00dd0cb-ef27-428f-b926-21396cb38b43" containerName="main"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142699    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00dd0cb-ef27-428f-b926-21396cb38b43" containerName="main"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142713    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ed16f6d6-d892-4603-a936-40f5e7ca4e73" containerName="payload-pre-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142720    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed16f6d6-d892-4603-a936-40f5e7ca4e73" containerName="payload-pre-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142749    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9fffc3d5-3d17-471e-8906-5a94ed21d14c" containerName="payload-pre-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142755    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fffc3d5-3d17-471e-8906-5a94ed21d14c" containerName="payload-pre-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142941    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00dd0cb-ef27-428f-b926-21396cb38b43" containerName="main"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142958    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed16f6d6-d892-4603-a936-40f5e7ca4e73" containerName="payload-pre-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142974    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fffc3d5-3d17-471e-8906-5a94ed21d14c" containerName="payload-pre-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.142986    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d86bb9-ee25-410c-99d3-adc646178897" containerName="payload-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143001    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6583710-32ad-4a57-9935-cb8953eaea21" containerName="payload-processing"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143010    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2befaa-bf7f-4806-9400-f40c87ab741c" containerName="maas-api"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143020    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143031    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace83d08-f3da-4b9d-abfd-e8050fa828a5" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143041    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8713c92b-ecbc-4b24-82b7-055b9dc84386" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143054    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7266464-203d-484f-8c3c-265e27639945" containerName="istio-proxy"
Nov 03 11:23:41.143048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143063    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cac8cf-3549-470b-b998-5cf85a1343bc" containerName="payload-pre-processing"
Nov 03 11:23:41.144685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143072    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6934784-f140-4ebb-af7d-b418d1238e93" containerName="main"
Nov 03 11:23:41.144685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143080    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="fccacc68-1e07-47c4-9878-6bf1dcafbd5b" containerName="istio-proxy"
Nov 03 11:23:41.144685 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.143088    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="db52f8d3-88ff-4cd9-91fc-9ac95ad82c48" containerName="istio-proxy"
Nov 03 11:23:41.150014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.149430    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.161368 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.157052    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-049d3b4c-openshift-default-dockercfg-fj9cz\""
Nov 03 11:23:41.161368 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.160729    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"]
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291273    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-socket\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291324    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-certs\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291413    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f308503-efc9-4f8d-9310-02616550cb70-istiod-ca-cert\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291453    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g877b\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-kube-api-access-g877b\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291526    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-istio-token\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291607    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-credential-socket\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291639    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-envoy\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291678    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-data\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.300309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.291709    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f308503-efc9-4f8d-9310-02616550cb70-istio-podinfo\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443255    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-socket\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443307    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-certs\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443411    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f308503-efc9-4f8d-9310-02616550cb70-istiod-ca-cert\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443449    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g877b\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-kube-api-access-g877b\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443531    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-istio-token\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443635    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-credential-socket\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443676    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-envoy\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443724    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-data\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.445997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.443771    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f308503-efc9-4f8d-9310-02616550cb70-istio-podinfo\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.463748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.461892    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-certs\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.463748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.462344    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-socket\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.463748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.462750    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-credential-socket\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.488978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.477092    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f308503-efc9-4f8d-9310-02616550cb70-istiod-ca-cert\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.488978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.477472    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-data\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.578009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.577323    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-envoy\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.578009 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.577891    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f308503-efc9-4f8d-9310-02616550cb70-istio-podinfo\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.598898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.578522    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g877b\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-kube-api-access-g877b\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.598898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.594453    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-istio-token\") pod \"e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") " pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:41.776226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:41.776133    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:42.068173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:42.068143    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"]
Nov 03 11:23:42.068900 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:23:42.068872    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f308503_efc9_4f8d_9310_02616550cb70.slice/crio-71fe532ec1972f685d9a9d2584c12dc1af91953ac8e63adbb1db495c8836022f WatchSource:0}: Error finding container 71fe532ec1972f685d9a9d2584c12dc1af91953ac8e63adbb1db495c8836022f: Status 404 returned error can't find the container with id 71fe532ec1972f685d9a9d2584c12dc1af91953ac8e63adbb1db495c8836022f
Nov 03 11:23:42.072886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:42.072848    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:23:42.072983 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:42.072952    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:23:42.073026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:42.072993    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:23:42.876071 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:42.875988    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" event={"ID":"6f308503-efc9-4f8d-9310-02616550cb70","Type":"ContainerStarted","Data":"ff4431e8ff46e4b00d17f8bb3e5a65e7f59ed7a5c353a60056bb7355abd91785"}
Nov 03 11:23:42.876803 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:42.876767    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" event={"ID":"6f308503-efc9-4f8d-9310-02616550cb70","Type":"ContainerStarted","Data":"71fe532ec1972f685d9a9d2584c12dc1af91953ac8e63adbb1db495c8836022f"}
Nov 03 11:23:42.907382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:42.907331    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" podStartSLOduration=1.907316521 podStartE2EDuration="1.907316521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:23:42.902786477 +0000 UTC m=+3266.807640919" watchObservedRunningTime="2026-11-03 11:23:42.907316521 +0000 UTC m=+3266.812170962"
Nov 03 11:23:43.194328 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:23:43.194243    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:23:43.776549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:43.776514    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:43.782938 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:43.782894    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:43.884891 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:43.884860    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:43.885870 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:43.885851    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:23:44.900254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.900045    5393 generic.go:356] "Generic (PLEG): container finished" podID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerID="faf88e517aa6c7af871e0cf7ace0817e08152da08b83df45b0133c8a9618fa3f" exitCode=0
Nov 03 11:23:44.900668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.900450    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" event={"ID":"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6","Type":"ContainerDied","Data":"faf88e517aa6c7af871e0cf7ace0817e08152da08b83df45b0133c8a9618fa3f"}
Nov 03 11:23:44.900668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.900577    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5" event={"ID":"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6","Type":"ContainerDied","Data":"d886594a6401fdf33cb7e104f13bc4216f13e202b1e0510f4b67003761ecd429"}
Nov 03 11:23:44.900668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.900616    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d886594a6401fdf33cb7e104f13bc4216f13e202b1e0510f4b67003761ecd429"
Nov 03 11:23:44.909476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.909445    5393 generic.go:356] "Generic (PLEG): container finished" podID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerID="0f4f0e8250b3dbabad9cbc47857acf47799aafe5e308bf06e54479f36d933be4" exitCode=0
Nov 03 11:23:44.909631 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.909499    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" event={"ID":"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4","Type":"ContainerDied","Data":"0f4f0e8250b3dbabad9cbc47857acf47799aafe5e308bf06e54479f36d933be4"}
Nov 03 11:23:44.909631 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.909554    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp" event={"ID":"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4","Type":"ContainerDied","Data":"49719efc81407424141d41f3284528b137687793529021857815673f1f246626"}
Nov 03 11:23:44.909631 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.909578    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49719efc81407424141d41f3284528b137687793529021857815673f1f246626"
Nov 03 11:23:44.913020 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.913003    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:23:44.916971 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:44.916953    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:23:45.040559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040524    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-token\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-token\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.040732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040635    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-socket\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.040732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040687    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-socket\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.040814 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040755    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-certs\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.040847 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040821    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-data\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.040899 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040871    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-podinfo\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.041022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040945    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-kube-api-access-jgqzl\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-kube-api-access-jgqzl\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.041022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040992    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-token\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-token\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.041022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.040997    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-socket" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.041196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041020    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-socket" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.041196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041071    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-certs\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.041196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041153    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-certs" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.041196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041166    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-data" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.041389 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041350    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-certs" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.041619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041595    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-envoy\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.041707 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041669    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-credential-socket\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.041780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041748    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-envoy\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.041839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.041830    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istiod-ca-cert\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.042113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.042012    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-credential-socket" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.042113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.042090    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-kube-api-access-bl5f9\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-kube-api-access-bl5f9\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.042262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.042145    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-podinfo\") pod \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\" (UID: \"3b77182d-ef2a-43e8-a7d8-3c5a790bccd6\") "
Nov 03 11:23:45.042262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.042191    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istiod-ca-cert\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.042262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.042226    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istiod-ca-cert" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:45.042446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.042276    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-data\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.042446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.042316    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-credential-socket\") pod \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\" (UID: \"4f66eaa5-f85d-4a53-88c6-c3fc84a676e4\") "
Nov 03 11:23:45.043437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043377    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.043437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043407    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.043437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043421    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.043437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043433    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.043698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043448    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.043698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043462    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.043698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043478    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.043698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043618    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-kube-api-access-jgqzl" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "kube-api-access-jgqzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:45.043934 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043804    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-podinfo" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:23:45.043934 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043849    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-credential-socket" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.043934 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.043880    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-token" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:45.044286 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.044256    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-data" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.044352 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.044283    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istiod-ca-cert" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:23:45.044650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.044631    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-token" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:45.044887 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.044870    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-envoy" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.045027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.045008    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-kube-api-access-bl5f9" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "kube-api-access-bl5f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:23:45.045472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.045455    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-envoy" pod "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" (UID: "4f66eaa5-f85d-4a53-88c6-c3fc84a676e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:23:45.045875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.045858    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-podinfo" pod "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" (UID: "3b77182d-ef2a-43e8-a7d8-3c5a790bccd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:23:45.145511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145425    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145457    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145467    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145476    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145486    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgqzl\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-kube-api-access-jgqzl\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145496    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145505    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145513    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145905 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145522    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5f9\" (UniqueName: \"kubernetes.io/projected/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-kube-api-access-bl5f9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145905 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145530    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.145905 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.145539    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:23:45.918536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.918442    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"
Nov 03 11:23:45.919005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.918755    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"
Nov 03 11:23:45.947885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.947853    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"]
Nov 03 11:23:45.952883 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.952857    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-xd2m5"]
Nov 03 11:23:45.972550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.972516    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"]
Nov 03 11:23:45.979305 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:45.979270    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c0a748-openshift-default-6f4f79d795-r5nbp"]
Nov 03 11:23:46.233293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:46.233219    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" path="/var/lib/kubelet/pods/3b77182d-ef2a-43e8-a7d8-3c5a790bccd6/volumes"
Nov 03 11:23:46.233713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:23:46.233699    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" path="/var/lib/kubelet/pods/4f66eaa5-f85d-4a53-88c6-c3fc84a676e4/volumes"
Nov 03 11:24:13.191254 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:24:13.191209    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:24:18.515887 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:18.515860    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:24:18.526416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:18.526392    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:24:38.514412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:38.514380    5393 generic.go:356] "Generic (PLEG): container finished" podID="6f308503-efc9-4f8d-9310-02616550cb70" containerID="ff4431e8ff46e4b00d17f8bb3e5a65e7f59ed7a5c353a60056bb7355abd91785" exitCode=0
Nov 03 11:24:38.514875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:38.514452    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" event={"ID":"6f308503-efc9-4f8d-9310-02616550cb70","Type":"ContainerDied","Data":"ff4431e8ff46e4b00d17f8bb3e5a65e7f59ed7a5c353a60056bb7355abd91785"}
Nov 03 11:24:38.515047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:38.515031    5393 scope.go:122] "RemoveContainer" containerID="ff4431e8ff46e4b00d17f8bb3e5a65e7f59ed7a5c353a60056bb7355abd91785"
Nov 03 11:24:38.521535 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:38.521502    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:24:38.521662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:38.521576    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:24:38.521662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:38.521609    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:24:39.525608 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:39.525569    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" event={"ID":"6f308503-efc9-4f8d-9310-02616550cb70","Type":"ContainerStarted","Data":"13650c70eaa7d227c207e00e266247c611d72b0b7aedbda6217295a15190c6e9"}
Nov 03 11:24:39.777271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:39.777183    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:24:39.778327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:39.778299    5393 patch_prober.go:29] interesting pod/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 03 11:24:39.778466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:39.778352    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 03 11:24:40.776744 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:40.776701    5393 patch_prober.go:29] interesting pod/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 03 11:24:40.777402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:40.776779    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 03 11:24:41.777486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:41.777449    5393 patch_prober.go:29] interesting pod/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 03 11:24:41.777953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:41.777509    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 03 11:24:42.776496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:42.776459    5393 patch_prober.go:29] interesting pod/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 03 11:24:42.776683 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:42.776523    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 03 11:24:43.216003 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:24:43.215888    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:24:43.776679 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:43.776642    5393 patch_prober.go:29] interesting pod/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 03 11:24:43.776885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:43.776699    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 03 11:24:43.905945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:43.905039    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:24:44.780614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:44.780575    5393 patch_prober.go:29] interesting pod/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 03 11:24:44.781282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:44.781249    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 03 11:24:45.782202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:45.782166    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:24:45.783595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:45.783563    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:24:49.043941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:49.043047    5393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-74hrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 11:24:49.043941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:24:49.043131    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-74hrv" podUID="55911ac4-1644-448a-bf1e-056207cdb3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:25:13.192676 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:25:13.192636    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:25:14.561937 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.561882    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"]
Nov 03 11:25:14.562791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.562769    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy"
Nov 03 11:25:14.562869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.562792    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy"
Nov 03 11:25:14.562869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.562824    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy"
Nov 03 11:25:14.562869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.562830    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy"
Nov 03 11:25:14.563054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.562987    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b77182d-ef2a-43e8-a7d8-3c5a790bccd6" containerName="istio-proxy"
Nov 03 11:25:14.563054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.563006    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f66eaa5-f85d-4a53-88c6-c3fc84a676e4" containerName="istio-proxy"
Nov 03 11:25:14.567491 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.567467    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.580078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.580051    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-9c53b2\"/\"openshift-service-ca.crt\""
Nov 03 11:25:14.580381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.580363    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-9c53b2\"/\"default-dockercfg-m927c\""
Nov 03 11:25:14.580548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.580534    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-9c53b2\"/\"sub-test-model-9c53b2-kserve-self-signed-certs\""
Nov 03 11:25:14.581962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.581939    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-9c53b2\"/\"kube-root-ca.crt\""
Nov 03 11:25:14.594107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.594077    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"]
Nov 03 11:25:14.691797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.691762    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-dshm\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.692023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.691896    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-model-cache\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.692023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.691995    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4694\" (UniqueName: \"kubernetes.io/projected/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-kube-api-access-q4694\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.692023 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.692023    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-home\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.692162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.692038    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tls-certs\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.692162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.692057    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tmp-dir\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.797027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.795972    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-dshm\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.797027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.796111    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-model-cache\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.797027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.796166    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4694\" (UniqueName: \"kubernetes.io/projected/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-kube-api-access-q4694\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.797027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.796207    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-home\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.797027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.796231    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tls-certs\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.797027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.796261    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tmp-dir\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.797027 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.796951    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tmp-dir\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.800940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.797870    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-model-cache\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.809949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.807196    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-home\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.841986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.839907    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-dshm\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.841986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.840485    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tls-certs\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.866946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.864654    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4694\" (UniqueName: \"kubernetes.io/projected/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-kube-api-access-q4694\") pod \"sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") " pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:14.900945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:14.900666    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:15.227629 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:15.227603    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"]
Nov 03 11:25:15.228274 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:25:15.228243    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod279ea1d2_0719_4d7f_bcf8_99f9d8f19270.slice/crio-574cfc4b921bbd578726260dad8008c8258f7191557e658ea21ac798fb1964c2 WatchSource:0}: Error finding container 574cfc4b921bbd578726260dad8008c8258f7191557e658ea21ac798fb1964c2: Status 404 returned error can't find the container with id 574cfc4b921bbd578726260dad8008c8258f7191557e658ea21ac798fb1964c2
Nov 03 11:25:15.952467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:15.952427    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966" event={"ID":"279ea1d2-0719-4d7f-bcf8-99f9d8f19270","Type":"ContainerStarted","Data":"53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4"}
Nov 03 11:25:15.952994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:15.952487    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966" event={"ID":"279ea1d2-0719-4d7f-bcf8-99f9d8f19270","Type":"ContainerStarted","Data":"574cfc4b921bbd578726260dad8008c8258f7191557e658ea21ac798fb1964c2"}
Nov 03 11:25:15.971318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:15.971262    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966" podStartSLOduration=1.9712479630000002 podStartE2EDuration="1.971247963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:25:15.970111537 +0000 UTC m=+3359.874965992" watchObservedRunningTime="2026-11-03 11:25:15.971247963 +0000 UTC m=+3359.876102404"
Nov 03 11:25:16.901611 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:16.901575    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:24.906626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:24.906589    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:24.919529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:24.919500    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:25.058794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:25.058767    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:25:30.617439 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.617245    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"]
Nov 03 11:25:30.627644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.627619    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.630330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.630284    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"]
Nov 03 11:25:30.641675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.639596    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-ef1662\"/\"sub-test-model-ef1662-kserve-self-signed-certs\""
Nov 03 11:25:30.641675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.640486    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-ef1662\"/\"kube-root-ca.crt\""
Nov 03 11:25:30.641675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.640556    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-ef1662\"/\"openshift-service-ca.crt\""
Nov 03 11:25:30.641872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.641791    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-ef1662\"/\"default-dockercfg-mfv6n\""
Nov 03 11:25:30.711590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.711549    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-home\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.711796 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.711668    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnvx\" (UniqueName: \"kubernetes.io/projected/a07ec5c1-d744-43cc-96f1-de34b47a99fd-kube-api-access-bgnvx\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.711861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.711806    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tmp-dir\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.711968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.711856    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-dshm\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.712190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.712168    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-model-cache\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.712287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.712270    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tls-certs\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.814549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.814508    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-dshm\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.815081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.815055    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-model-cache\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.815278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.815260    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tls-certs\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.815431 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.815417    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-home\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.815554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.815540    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnvx\" (UniqueName: \"kubernetes.io/projected/a07ec5c1-d744-43cc-96f1-de34b47a99fd-kube-api-access-bgnvx\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.815709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.815696    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tmp-dir\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.816362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.816340    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tmp-dir\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.817392 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.817370    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-home\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.831159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.831134    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-model-cache\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.844949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.837946    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tls-certs\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.844949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.838808    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnvx\" (UniqueName: \"kubernetes.io/projected/a07ec5c1-d744-43cc-96f1-de34b47a99fd-kube-api-access-bgnvx\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.876010 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.872969    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-dshm\") pod \"sub-test-model-ef1662-kserve-5bb4477b49-cqvhq\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") " pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:30.951433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:30.950074    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:31.281664 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:25:31.281635    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07ec5c1_d744_43cc_96f1_de34b47a99fd.slice/crio-5f17d699b106f5f3b80b2ba6de9a8b13139958c2c0dd2aec9bc8cdc50a50eae7 WatchSource:0}: Error finding container 5f17d699b106f5f3b80b2ba6de9a8b13139958c2c0dd2aec9bc8cdc50a50eae7: Status 404 returned error can't find the container with id 5f17d699b106f5f3b80b2ba6de9a8b13139958c2c0dd2aec9bc8cdc50a50eae7
Nov 03 11:25:31.283103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:31.283066    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"]
Nov 03 11:25:32.114831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:32.114795    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq" event={"ID":"a07ec5c1-d744-43cc-96f1-de34b47a99fd","Type":"ContainerStarted","Data":"936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7"}
Nov 03 11:25:32.115280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:32.114842    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq" event={"ID":"a07ec5c1-d744-43cc-96f1-de34b47a99fd","Type":"ContainerStarted","Data":"5f17d699b106f5f3b80b2ba6de9a8b13139958c2c0dd2aec9bc8cdc50a50eae7"}
Nov 03 11:25:32.131775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:32.131719    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq" podStartSLOduration=2.131705372 podStartE2EDuration="2.131705372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:25:32.131424141 +0000 UTC m=+3376.036278583" watchObservedRunningTime="2026-11-03 11:25:32.131705372 +0000 UTC m=+3376.036559812"
Nov 03 11:25:32.950833 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:32.950780    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:37.813472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:37.812909    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"]
Nov 03 11:25:40.950540 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:40.950509    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:40.965458 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:40.965418    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:41.231456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:41.231369    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:25:43.192342 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:25:43.192300    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:25:48.872639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:48.872583    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"]
Nov 03 11:25:52.836899 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:52.836847    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm" podUID="b9b97e11-284c-483d-a6b3-a50073df6247" containerName="main" containerID="cri-o://d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a" gracePeriod=45
Nov 03 11:25:52.981705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:52.981663    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm" podUID="b9b97e11-284c-483d-a6b3-a50073df6247" containerName="main" probeResult="failure" output="Get \"https://10.134.1.39:8000/ready\": dial tcp 10.134.1.39:8000: connect: connection refused"
Nov 03 11:25:53.180634 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.180609    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:25:53.239710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.239678    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-dshm\") pod \"b9b97e11-284c-483d-a6b3-a50073df6247\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") "
Nov 03 11:25:53.239963 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.239819    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-home\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-home\") pod \"b9b97e11-284c-483d-a6b3-a50073df6247\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") "
Nov 03 11:25:53.239963 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.239898    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-model-cache\") pod \"b9b97e11-284c-483d-a6b3-a50073df6247\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") "
Nov 03 11:25:53.240134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.240069    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b97e11-284c-483d-a6b3-a50073df6247-kube-api-access-rqrf2\" (UniqueName: \"kubernetes.io/projected/b9b97e11-284c-483d-a6b3-a50073df6247-kube-api-access-rqrf2\") pod \"b9b97e11-284c-483d-a6b3-a50073df6247\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") "
Nov 03 11:25:53.240204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.240130    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-home" pod "b9b97e11-284c-483d-a6b3-a50073df6247" (UID: "b9b97e11-284c-483d-a6b3-a50073df6247"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:25:53.240204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.240180    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b97e11-284c-483d-a6b3-a50073df6247-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b97e11-284c-483d-a6b3-a50073df6247-tls-certs\") pod \"b9b97e11-284c-483d-a6b3-a50073df6247\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") "
Nov 03 11:25:53.240204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.240186    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-model-cache" pod "b9b97e11-284c-483d-a6b3-a50073df6247" (UID: "b9b97e11-284c-483d-a6b3-a50073df6247"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:25:53.240385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.240205    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-tmp-dir\") pod \"b9b97e11-284c-483d-a6b3-a50073df6247\" (UID: \"b9b97e11-284c-483d-a6b3-a50073df6247\") "
Nov 03 11:25:53.240496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.240475    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-tmp-dir" pod "b9b97e11-284c-483d-a6b3-a50073df6247" (UID: "b9b97e11-284c-483d-a6b3-a50073df6247"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:25:53.241106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.241079    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.241367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.241111    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.241367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.241125    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.243216 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.243175    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b97e11-284c-483d-a6b3-a50073df6247-tls-certs" pod "b9b97e11-284c-483d-a6b3-a50073df6247" (UID: "b9b97e11-284c-483d-a6b3-a50073df6247"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:25:53.243337 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.243301    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-dshm" pod "b9b97e11-284c-483d-a6b3-a50073df6247" (UID: "b9b97e11-284c-483d-a6b3-a50073df6247"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:25:53.243337 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.243324    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b97e11-284c-483d-a6b3-a50073df6247-kube-api-access-rqrf2" pod "b9b97e11-284c-483d-a6b3-a50073df6247" (UID: "b9b97e11-284c-483d-a6b3-a50073df6247"). InnerVolumeSpecName "kube-api-access-rqrf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:25:53.319960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.319899    5393 generic.go:356] "Generic (PLEG): container finished" podID="b9b97e11-284c-483d-a6b3-a50073df6247" containerID="d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a" exitCode=0
Nov 03 11:25:53.320133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.319968    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"
Nov 03 11:25:53.320133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.319979    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm" event={"ID":"b9b97e11-284c-483d-a6b3-a50073df6247","Type":"ContainerDied","Data":"d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a"}
Nov 03 11:25:53.320133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.320025    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm" event={"ID":"b9b97e11-284c-483d-a6b3-a50073df6247","Type":"ContainerDied","Data":"67f485b016103f4f9fee041ee704deabac729fdc25f551a1d84490fab266e945"}
Nov 03 11:25:53.320133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.320042    5393 scope.go:122] "RemoveContainer" containerID="d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a"
Nov 03 11:25:53.332437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.332408    5393 scope.go:122] "RemoveContainer" containerID="d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a"
Nov 03 11:25:53.332779 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:25:53.332753    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a\": container with ID starting with d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a not found: ID does not exist" containerID="d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a"
Nov 03 11:25:53.332856 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.332789    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a"} err="failed to get container status \"d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a\": rpc error: code = NotFound desc = could not find container \"d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a\": container with ID starting with d36098320363367d1b7b048aba493b363c1d790145a2a28e8ae7f0c54fabec3a not found: ID does not exist"
Nov 03 11:25:53.343251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.343221    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrf2\" (UniqueName: \"kubernetes.io/projected/b9b97e11-284c-483d-a6b3-a50073df6247-kube-api-access-rqrf2\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.343377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.343252    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9b97e11-284c-483d-a6b3-a50073df6247-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.343377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.343288    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9b97e11-284c-483d-a6b3-a50073df6247-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:25:53.346952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.346928    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"]
Nov 03 11:25:53.351562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:53.351538    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b13404/test-model-b13404-kserve-77f48ffc55-5vvhm"]
Nov 03 11:25:54.233683 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:25:54.233653    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b97e11-284c-483d-a6b3-a50073df6247" path="/var/lib/kubelet/pods/b9b97e11-284c-483d-a6b3-a50073df6247/volumes"
Nov 03 11:26:02.907057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:02.907019    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"]
Nov 03 11:26:02.907775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:02.907758    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b9b97e11-284c-483d-a6b3-a50073df6247" containerName="main"
Nov 03 11:26:02.907775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:02.907774    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b97e11-284c-483d-a6b3-a50073df6247" containerName="main"
Nov 03 11:26:02.908040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:02.908026    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b97e11-284c-483d-a6b3-a50073df6247" containerName="main"
Nov 03 11:26:02.913190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:02.913161    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:02.916871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:02.916842    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"]
Nov 03 11:26:02.918208 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:02.918189    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:26:02.918322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:02.918269    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:26:03.060390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.060353    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.060619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.060596    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.060667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.060638    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ng4d\" (UniqueName: \"kubernetes.io/projected/bb4ca33d-3407-4793-97be-a091fcae22cb-kube-api-access-9ng4d\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.161950 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.161833    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.162119 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.162036    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.162119 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.162065    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ng4d\" (UniqueName: \"kubernetes.io/projected/bb4ca33d-3407-4793-97be-a091fcae22cb-kube-api-access-9ng4d\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.162581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.162557    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.164630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.164605    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.170118 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.170093    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ng4d\" (UniqueName: \"kubernetes.io/projected/bb4ca33d-3407-4793-97be-a091fcae22cb-kube-api-access-9ng4d\") pod \"maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.233225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.233194    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:03.520640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.520605    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"]
Nov 03 11:26:03.520858 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:26:03.520830    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4ca33d_3407_4793_97be_a091fcae22cb.slice/crio-70b1ff2379cf24614bac62c54e39e686cedb71b632c8415651f8bf19bcf79998 WatchSource:0}: Error finding container 70b1ff2379cf24614bac62c54e39e686cedb71b632c8415651f8bf19bcf79998: Status 404 returned error can't find the container with id 70b1ff2379cf24614bac62c54e39e686cedb71b632c8415651f8bf19bcf79998
Nov 03 11:26:03.927331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:03.927272    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27" podUID="ea7380ea-3664-469f-a0d4-c947bdecb710" containerName="main" containerID="cri-o://485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072" gracePeriod=45
Nov 03 11:26:04.262976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.256878    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:26:04.385557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.385516    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-dshm\") pod \"ea7380ea-3664-469f-a0d4-c947bdecb710\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") "
Nov 03 11:26:04.385760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.385587    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-home\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-home\") pod \"ea7380ea-3664-469f-a0d4-c947bdecb710\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") "
Nov 03 11:26:04.385760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.385721    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-model-cache\") pod \"ea7380ea-3664-469f-a0d4-c947bdecb710\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") "
Nov 03 11:26:04.385855 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.385802    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7380ea-3664-469f-a0d4-c947bdecb710-kube-api-access-xvk7t\" (UniqueName: \"kubernetes.io/projected/ea7380ea-3664-469f-a0d4-c947bdecb710-kube-api-access-xvk7t\") pod \"ea7380ea-3664-469f-a0d4-c947bdecb710\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") "
Nov 03 11:26:04.385993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.385971    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-tmp-dir\") pod \"ea7380ea-3664-469f-a0d4-c947bdecb710\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") "
Nov 03 11:26:04.386115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.386099    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea7380ea-3664-469f-a0d4-c947bdecb710-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7380ea-3664-469f-a0d4-c947bdecb710-tls-certs\") pod \"ea7380ea-3664-469f-a0d4-c947bdecb710\" (UID: \"ea7380ea-3664-469f-a0d4-c947bdecb710\") "
Nov 03 11:26:04.388567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.388414    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-model-cache" pod "ea7380ea-3664-469f-a0d4-c947bdecb710" (UID: "ea7380ea-3664-469f-a0d4-c947bdecb710"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:04.388567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.388617    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-home" pod "ea7380ea-3664-469f-a0d4-c947bdecb710" (UID: "ea7380ea-3664-469f-a0d4-c947bdecb710"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:04.389079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.388833    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-tmp-dir" pod "ea7380ea-3664-469f-a0d4-c947bdecb710" (UID: "ea7380ea-3664-469f-a0d4-c947bdecb710"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:04.390464 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.390440    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-dshm" pod "ea7380ea-3664-469f-a0d4-c947bdecb710" (UID: "ea7380ea-3664-469f-a0d4-c947bdecb710"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:04.390577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.390518    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7380ea-3664-469f-a0d4-c947bdecb710-tls-certs" pod "ea7380ea-3664-469f-a0d4-c947bdecb710" (UID: "ea7380ea-3664-469f-a0d4-c947bdecb710"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:26:04.391155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.391135    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7380ea-3664-469f-a0d4-c947bdecb710-kube-api-access-xvk7t" pod "ea7380ea-3664-469f-a0d4-c947bdecb710" (UID: "ea7380ea-3664-469f-a0d4-c947bdecb710"). InnerVolumeSpecName "kube-api-access-xvk7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:04.445975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.445870    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6" event={"ID":"bb4ca33d-3407-4793-97be-a091fcae22cb","Type":"ContainerStarted","Data":"de47b4bbf4a91b1c47b98936759fd7df96246e9a5295b68971171565d2b08747"}
Nov 03 11:26:04.445975 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.445948    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6" event={"ID":"bb4ca33d-3407-4793-97be-a091fcae22cb","Type":"ContainerStarted","Data":"70b1ff2379cf24614bac62c54e39e686cedb71b632c8415651f8bf19bcf79998"}
Nov 03 11:26:04.447556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.447526    5393 generic.go:356] "Generic (PLEG): container finished" podID="ea7380ea-3664-469f-a0d4-c947bdecb710" containerID="485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072" exitCode=0
Nov 03 11:26:04.447702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.447563    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"
Nov 03 11:26:04.447702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.447577    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27" event={"ID":"ea7380ea-3664-469f-a0d4-c947bdecb710","Type":"ContainerDied","Data":"485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072"}
Nov 03 11:26:04.447702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.447603    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27" event={"ID":"ea7380ea-3664-469f-a0d4-c947bdecb710","Type":"ContainerDied","Data":"e04712e14f1bebc9b9d07879dcc8eca8b6209701044c6e6e76425a92b5dd2b4e"}
Nov 03 11:26:04.447702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.447621    5393 scope.go:122] "RemoveContainer" containerID="485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072"
Nov 03 11:26:04.459392 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.459368    5393 scope.go:122] "RemoveContainer" containerID="485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072"
Nov 03 11:26:04.459686 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:26:04.459667    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072\": container with ID starting with 485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072 not found: ID does not exist" containerID="485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072"
Nov 03 11:26:04.459733 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.459693    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072"} err="failed to get container status \"485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072\": rpc error: code = NotFound desc = could not find container \"485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072\": container with ID starting with 485a29448763c617be6119d3eb72463608d6e45cf4ecd978f6977628365e1072 not found: ID does not exist"
Nov 03 11:26:04.467545 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.467499    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6" podStartSLOduration=2.467486636 podStartE2EDuration="2.467486636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:26:04.463226555 +0000 UTC m=+3408.368081008" watchObservedRunningTime="2026-11-03 11:26:04.467486636 +0000 UTC m=+3408.372341071"
Nov 03 11:26:04.478320 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.478296    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"]
Nov 03 11:26:04.481955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.481933    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27"]
Nov 03 11:26:04.488568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.488543    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:04.488568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.488567    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:04.488758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.488576    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:04.488758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.488648    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvk7t\" (UniqueName: \"kubernetes.io/projected/ea7380ea-3664-469f-a0d4-c947bdecb710-kube-api-access-xvk7t\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:04.488758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.488701    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea7380ea-3664-469f-a0d4-c947bdecb710-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:04.488964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:04.488824    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea7380ea-3664-469f-a0d4-c947bdecb710-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:05.206419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:05.206372    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9143e1/test-model-9143e1-kserve-7bb7d77bc7-hcl27" podUID="ea7380ea-3664-469f-a0d4-c947bdecb710" containerName="main" probeResult="failure" output="Get \"https://10.134.1.40:8000/ready\": context deadline exceeded"
Nov 03 11:26:05.509058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:05.508902    5393 generic.go:356] "Generic (PLEG): container finished" podID="bb4ca33d-3407-4793-97be-a091fcae22cb" containerID="de47b4bbf4a91b1c47b98936759fd7df96246e9a5295b68971171565d2b08747" exitCode=0
Nov 03 11:26:05.509498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:05.509465    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6" event={"ID":"bb4ca33d-3407-4793-97be-a091fcae22cb","Type":"ContainerDied","Data":"de47b4bbf4a91b1c47b98936759fd7df96246e9a5295b68971171565d2b08747"}
Nov 03 11:26:06.233019 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.232991    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7380ea-3664-469f-a0d4-c947bdecb710" path="/var/lib/kubelet/pods/ea7380ea-3664-469f-a0d4-c947bdecb710/volumes"
Nov 03 11:26:06.696186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.696155    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:06.732496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.732458    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-xks-ca\") pod \"bb4ca33d-3407-4793-97be-a091fcae22cb\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") "
Nov 03 11:26:06.733330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.733304    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4ca33d-3407-4793-97be-a091fcae22cb-kube-api-access-9ng4d\" (UniqueName: \"kubernetes.io/projected/bb4ca33d-3407-4793-97be-a091fcae22cb-kube-api-access-9ng4d\") pod \"bb4ca33d-3407-4793-97be-a091fcae22cb\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") "
Nov 03 11:26:06.733464 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.733344    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-openshift-ca\") pod \"bb4ca33d-3407-4793-97be-a091fcae22cb\" (UID: \"bb4ca33d-3407-4793-97be-a091fcae22cb\") "
Nov 03 11:26:06.733779 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.733755    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-openshift-ca" pod "bb4ca33d-3407-4793-97be-a091fcae22cb" (UID: "bb4ca33d-3407-4793-97be-a091fcae22cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:06.734125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.734092    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:06.735530 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.735500    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-xks-ca" pod "bb4ca33d-3407-4793-97be-a091fcae22cb" (UID: "bb4ca33d-3407-4793-97be-a091fcae22cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:26:06.736197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.736168    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4ca33d-3407-4793-97be-a091fcae22cb-kube-api-access-9ng4d" pod "bb4ca33d-3407-4793-97be-a091fcae22cb" (UID: "bb4ca33d-3407-4793-97be-a091fcae22cb"). InnerVolumeSpecName "kube-api-access-9ng4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:06.835805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.835722    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ng4d\" (UniqueName: \"kubernetes.io/projected/bb4ca33d-3407-4793-97be-a091fcae22cb-kube-api-access-9ng4d\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:06.835805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:06.835762    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb4ca33d-3407-4793-97be-a091fcae22cb-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:07.540350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:07.540321    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"
Nov 03 11:26:07.540350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:07.540345    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6" event={"ID":"bb4ca33d-3407-4793-97be-a091fcae22cb","Type":"ContainerDied","Data":"70b1ff2379cf24614bac62c54e39e686cedb71b632c8415651f8bf19bcf79998"}
Nov 03 11:26:07.540958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:07.540396    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70b1ff2379cf24614bac62c54e39e686cedb71b632c8415651f8bf19bcf79998"
Nov 03 11:26:13.191729 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:26:13.191692    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:26:15.829944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:15.829843    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"]
Nov 03 11:26:16.062148 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:16.001597    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-049d3b4c-8kss6"]
Nov 03 11:26:16.275340 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:16.275304    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4ca33d-3407-4793-97be-a091fcae22cb" path="/var/lib/kubelet/pods/bb4ca33d-3407-4793-97be-a091fcae22cb/volumes"
Nov 03 11:26:18.301889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:18.301857    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"]
Nov 03 11:26:18.302646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:18.302616    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy" containerID="cri-o://55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16" gracePeriod=30
Nov 03 11:26:22.052941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:22.049965    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"]
Nov 03 11:26:22.084941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:22.083768    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-637e41cb-x697v"]
Nov 03 11:26:22.233499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:22.233468    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5630cb64-48b7-4aa0-8cf1-64f867be7f06" path="/var/lib/kubelet/pods/5630cb64-48b7-4aa0-8cf1-64f867be7f06/volumes"
Nov 03 11:26:22.601189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:22.601154    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"]
Nov 03 11:26:22.601439 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:22.601393    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy" containerID="cri-o://13650c70eaa7d227c207e00e266247c611d72b0b7aedbda6217295a15190c6e9" gracePeriod=30
Nov 03 11:26:23.613883 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.613855    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:26:23.692749 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.692711    5393 generic.go:356] "Generic (PLEG): container finished" podID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerID="55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16" exitCode=0
Nov 03 11:26:23.693007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.692767    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"
Nov 03 11:26:23.693007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.692804    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" event={"ID":"6a685fa6-5b4a-45bc-8877-815f154949d3","Type":"ContainerDied","Data":"55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16"}
Nov 03 11:26:23.693007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.692867    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl" event={"ID":"6a685fa6-5b4a-45bc-8877-815f154949d3","Type":"ContainerDied","Data":"2d87bccea3063bac53b06a86f3a7606ec8236c1531d5d5834e1ee223022598a7"}
Nov 03 11:26:23.693007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.692898    5393 scope.go:122] "RemoveContainer" containerID="55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16"
Nov 03 11:26:23.705035 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.705009    5393 scope.go:122] "RemoveContainer" containerID="55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16"
Nov 03 11:26:23.705373 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:26:23.705353    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16\": container with ID starting with 55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16 not found: ID does not exist" containerID="55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16"
Nov 03 11:26:23.705432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.705384    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16"} err="failed to get container status \"55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16\": rpc error: code = NotFound desc = could not find container \"55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16\": container with ID starting with 55b3d34d866e8d7f4d732ac3d69925383b08820db788d96e576c106ee84fac16 not found: ID does not exist"
Nov 03 11:26:23.761450 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.761403    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-certs\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.761667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.761519    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-socket\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.761667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.761573    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a685fa6-5b4a-45bc-8877-815f154949d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a685fa6-5b4a-45bc-8877-815f154949d3-istiod-ca-cert\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.761822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.761730    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-certs" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.762204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762181    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a685fa6-5b4a-45bc-8877-815f154949d3-istiod-ca-cert" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:23.762338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762303    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-kube-api-access-fqc8p\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-kube-api-access-fqc8p\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.762471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762366    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-token\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-token\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.762471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762443    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-socket" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.762471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762448    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-envoy\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.762673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762560    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-credential-socket\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.762673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762602    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-data\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.762673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762663    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-podinfo\") pod \"6a685fa6-5b4a-45bc-8877-815f154949d3\" (UID: \"6a685fa6-5b4a-45bc-8877-815f154949d3\") "
Nov 03 11:26:23.763054 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.762973    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-credential-socket" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.763693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.763668    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.763805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.763698    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.763805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.763712    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.763805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.763725    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a685fa6-5b4a-45bc-8877-815f154949d3-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.764088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.763965    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-data" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.765413 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.765386    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-envoy" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:23.765763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.765733    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-podinfo" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:26:23.765763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.765731    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-kube-api-access-fqc8p" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "kube-api-access-fqc8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:23.766325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.766304    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-token" pod "6a685fa6-5b4a-45bc-8877-815f154949d3" (UID: "6a685fa6-5b4a-45bc-8877-815f154949d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:23.864787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.864757    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.864787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.864785    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.864787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.864796    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.865086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.864805    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqc8p\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-kube-api-access-fqc8p\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:23.865086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:23.864815    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a685fa6-5b4a-45bc-8877-815f154949d3-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:24.018437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:24.018387    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"]
Nov 03 11:26:24.022491 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:24.022466    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b13404-openshift-default-7d5579f55-wmnxl"]
Nov 03 11:26:24.233644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:24.233569    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" path="/var/lib/kubelet/pods/6a685fa6-5b4a-45bc-8877-815f154949d3/volumes"
Nov 03 11:26:27.731524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.731488    5393 generic.go:356] "Generic (PLEG): container finished" podID="6f308503-efc9-4f8d-9310-02616550cb70" containerID="13650c70eaa7d227c207e00e266247c611d72b0b7aedbda6217295a15190c6e9" exitCode=0
Nov 03 11:26:27.731983 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.731537    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" event={"ID":"6f308503-efc9-4f8d-9310-02616550cb70","Type":"ContainerDied","Data":"13650c70eaa7d227c207e00e266247c611d72b0b7aedbda6217295a15190c6e9"}
Nov 03 11:26:27.731983 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.731608    5393 scope.go:122] "RemoveContainer" containerID="ff4431e8ff46e4b00d17f8bb3e5a65e7f59ed7a5c353a60056bb7355abd91785"
Nov 03 11:26:27.900297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.900269    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:26:27.928589 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.928547    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-credential-socket\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.928802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.928681    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-certs\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.928802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.928773    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-data\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.928989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.928813    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-kube-api-access-g877b\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-kube-api-access-g877b\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.928989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.928858    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-credential-socket" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:27.928989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.928896    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-envoy\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.929234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.928999    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-certs" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:27.929234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.929027    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-data" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:27.929234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.929035    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f308503-efc9-4f8d-9310-02616550cb70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f308503-efc9-4f8d-9310-02616550cb70-istiod-ca-cert\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.929234 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.929149    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-istio-token\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-istio-token\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.929408 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.929259    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-socket\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.929408 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.929312    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f308503-efc9-4f8d-9310-02616550cb70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f308503-efc9-4f8d-9310-02616550cb70-istio-podinfo\") pod \"6f308503-efc9-4f8d-9310-02616550cb70\" (UID: \"6f308503-efc9-4f8d-9310-02616550cb70\") "
Nov 03 11:26:27.929408 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.929328    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f308503-efc9-4f8d-9310-02616550cb70-istiod-ca-cert" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:27.929578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.929499    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-socket" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:27.930229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.930198    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:27.930367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.930231    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:27.930367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.930247    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f308503-efc9-4f8d-9310-02616550cb70-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:27.930367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.930262    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:27.930367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.930277    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:27.931570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.931547    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-istio-token" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:27.931679 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.931581    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-kube-api-access-g877b" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "kube-api-access-g877b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:27.931727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.931690    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-envoy" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:27.931844 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:27.931820    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f308503-efc9-4f8d-9310-02616550cb70-istio-podinfo" pod "6f308503-efc9-4f8d-9310-02616550cb70" (UID: "6f308503-efc9-4f8d-9310-02616550cb70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:26:28.031600 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.031564    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:28.031600 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.031597    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f308503-efc9-4f8d-9310-02616550cb70-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:28.031822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.031608    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g877b\" (UniqueName: \"kubernetes.io/projected/6f308503-efc9-4f8d-9310-02616550cb70-kube-api-access-g877b\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:28.031822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.031619    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f308503-efc9-4f8d-9310-02616550cb70-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:28.740390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.740363    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"
Nov 03 11:26:28.740876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.740411    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g" event={"ID":"6f308503-efc9-4f8d-9310-02616550cb70","Type":"ContainerDied","Data":"71fe532ec1972f685d9a9d2584c12dc1af91953ac8e63adbb1db495c8836022f"}
Nov 03 11:26:28.740876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.740468    5393 scope.go:122] "RemoveContainer" containerID="13650c70eaa7d227c207e00e266247c611d72b0b7aedbda6217295a15190c6e9"
Nov 03 11:26:28.762364 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.762329    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"]
Nov 03 11:26:28.767774 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.767746    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-049d3b4c-openshift-default-9b7d7b57c-xfs6g"]
Nov 03 11:26:28.896035 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:28.895999    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"]
Nov 03 11:26:30.233468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:30.233432    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f308503-efc9-4f8d-9310-02616550cb70" path="/var/lib/kubelet/pods/6f308503-efc9-4f8d-9310-02616550cb70/volumes"
Nov 03 11:26:33.896641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:33.896603    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v" podUID="5d297c0a-8957-4a5d-b652-4c11a35b9b0d" containerName="payload-processing" containerID="cri-o://a0f18cf25acd65a42cb762653b42675adfe843dc02736771a0e0ef7eb6794e5c" gracePeriod=25
Nov 03 11:26:41.062999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:41.062953    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v" podUID="5d297c0a-8957-4a5d-b652-4c11a35b9b0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:26:43.193221 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:26:43.193178    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:26:49.534709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:49.534661    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"]
Nov 03 11:26:49.535282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:49.534934    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy" containerID="cri-o://064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c" gracePeriod=30
Nov 03 11:26:51.062327 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:51.062287    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v" podUID="5d297c0a-8957-4a5d-b652-4c11a35b9b0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:26:52.205784 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:52.205686    5393 patch_prober.go:29] interesting pod/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:26:52.206523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:52.205869    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:26:54.109121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.109083    5393 generic.go:356] "Generic (PLEG): container finished" podID="5d297c0a-8957-4a5d-b652-4c11a35b9b0d" containerID="a0f18cf25acd65a42cb762653b42675adfe843dc02736771a0e0ef7eb6794e5c" exitCode=0
Nov 03 11:26:54.109513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.109136    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v" event={"ID":"5d297c0a-8957-4a5d-b652-4c11a35b9b0d","Type":"ContainerDied","Data":"a0f18cf25acd65a42cb762653b42675adfe843dc02736771a0e0ef7eb6794e5c"}
Nov 03 11:26:54.246514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.246487    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:26:54.321843 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.321805    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-kube-api-access-2dpwn\" (UniqueName: \"kubernetes.io/projected/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-kube-api-access-2dpwn\") pod \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\" (UID: \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\") "
Nov 03 11:26:54.322073 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.322044    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-plugins-config-volume\") pod \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\" (UID: \"5d297c0a-8957-4a5d-b652-4c11a35b9b0d\") "
Nov 03 11:26:54.322404 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.322381    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-plugins-config-volume" pod "5d297c0a-8957-4a5d-b652-4c11a35b9b0d" (UID: "5d297c0a-8957-4a5d-b652-4c11a35b9b0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:54.322844 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.322820    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.324230 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.324209    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-kube-api-access-2dpwn" pod "5d297c0a-8957-4a5d-b652-4c11a35b9b0d" (UID: "5d297c0a-8957-4a5d-b652-4c11a35b9b0d"). InnerVolumeSpecName "kube-api-access-2dpwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:54.424270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.424228    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dpwn\" (UniqueName: \"kubernetes.io/projected/5d297c0a-8957-4a5d-b652-4c11a35b9b0d-kube-api-access-2dpwn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.802210 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.802184    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:26:54.937321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937289    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-kube-api-access-nmlk8\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-kube-api-access-nmlk8\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.937480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937356    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-certs\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.937528 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937509    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-socket\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.937590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937576    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-envoy\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.937657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937634    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-certs" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:54.937768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937743    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-podinfo\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.937904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937783    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-token\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-token\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.937904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937782    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-socket" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:54.937904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937811    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9be7fd-4617-4579-a056-80f3bdb258cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9be7fd-4617-4579-a056-80f3bdb258cb-istiod-ca-cert\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.938109 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937958    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-credential-socket\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.938406 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.937989    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-data\") pod \"1d9be7fd-4617-4579-a056-80f3bdb258cb\" (UID: \"1d9be7fd-4617-4579-a056-80f3bdb258cb\") "
Nov 03 11:26:54.938551 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.938309    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-data" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:54.938655 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.938326    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-credential-socket" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:54.938733 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.938340    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9be7fd-4617-4579-a056-80f3bdb258cb-istiod-ca-cert" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:26:54.939200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.939177    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9be7fd-4617-4579-a056-80f3bdb258cb-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.939293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.939205    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.939293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.939219    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.939293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.939232    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.939293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.939246    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:54.939980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.939951    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-kube-api-access-nmlk8" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "kube-api-access-nmlk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:54.940322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.940304    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-token" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:26:54.940499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.940483    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-envoy" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:26:54.940596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:54.940581    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-podinfo" pod "1d9be7fd-4617-4579-a056-80f3bdb258cb" (UID: "1d9be7fd-4617-4579-a056-80f3bdb258cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:26:55.040714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.040676    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmlk8\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-kube-api-access-nmlk8\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:55.040714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.040716    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:55.040954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.040730    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:55.040954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.040743    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9be7fd-4617-4579-a056-80f3bdb258cb-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:26:55.119163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.119131    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"
Nov 03 11:26:55.119668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.119149    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v" event={"ID":"5d297c0a-8957-4a5d-b652-4c11a35b9b0d","Type":"ContainerDied","Data":"4fb5e3ff38201fce27a06ba1c18f8e38a48984a66b00c49c0695815ca383f665"}
Nov 03 11:26:55.119668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.119228    5393 scope.go:122] "RemoveContainer" containerID="a0f18cf25acd65a42cb762653b42675adfe843dc02736771a0e0ef7eb6794e5c"
Nov 03 11:26:55.126417 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.126390    5393 generic.go:356] "Generic (PLEG): container finished" podID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerID="064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c" exitCode=0
Nov 03 11:26:55.126546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.126441    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"
Nov 03 11:26:55.126546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.126433    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" event={"ID":"1d9be7fd-4617-4579-a056-80f3bdb258cb","Type":"ContainerDied","Data":"064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c"}
Nov 03 11:26:55.126546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.126507    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz" event={"ID":"1d9be7fd-4617-4579-a056-80f3bdb258cb","Type":"ContainerDied","Data":"4f4d97a1fc741822fafdebf21fd607d2d6c2ec457ae62f51ba1abb00546b6615"}
Nov 03 11:26:55.136627 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.136607    5393 scope.go:122] "RemoveContainer" containerID="064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c"
Nov 03 11:26:55.149616 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.149593    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"]
Nov 03 11:26:55.152057 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.152037    5393 scope.go:122] "RemoveContainer" containerID="919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910"
Nov 03 11:26:55.159828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.159803    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9143e1-bffbfb589-f687v"]
Nov 03 11:26:55.167600 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.167566    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"]
Nov 03 11:26:55.171147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.171082    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9143e1-openshift-default-5cbd6c78ff-f98lz"]
Nov 03 11:26:55.177818 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.177798    5393 scope.go:122] "RemoveContainer" containerID="064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c"
Nov 03 11:26:55.178176 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:26:55.178154    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c\": container with ID starting with 064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c not found: ID does not exist" containerID="064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c"
Nov 03 11:26:55.178235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.178186    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c"} err="failed to get container status \"064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c\": rpc error: code = NotFound desc = could not find container \"064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c\": container with ID starting with 064720b70b6968b7321e48606512193047f8f01267889a9abc1b92d2fc44bd3c not found: ID does not exist"
Nov 03 11:26:55.178235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.178208    5393 scope.go:122] "RemoveContainer" containerID="919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910"
Nov 03 11:26:55.178517 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:26:55.178495    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910\": container with ID starting with 919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910 not found: ID does not exist" containerID="919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910"
Nov 03 11:26:55.178578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:55.178523    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910"} err="failed to get container status \"919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910\": rpc error: code = NotFound desc = could not find container \"919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910\": container with ID starting with 919ca99bb62c2b509145b1fc54bc8fde129c84871165f5525650d64f377cb910 not found: ID does not exist"
Nov 03 11:26:56.234661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:56.234629    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" path="/var/lib/kubelet/pods/1d9be7fd-4617-4579-a056-80f3bdb258cb/volumes"
Nov 03 11:26:56.235121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:26:56.235097    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d297c0a-8957-4a5d-b652-4c11a35b9b0d" path="/var/lib/kubelet/pods/5d297c0a-8957-4a5d-b652-4c11a35b9b0d/volumes"
Nov 03 11:27:00.236730 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:00.236693    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"]
Nov 03 11:27:02.762440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:02.762401    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"]
Nov 03 11:27:02.763033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:02.762690    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h" podUID="a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" containerName="maas-api" containerID="cri-o://a2200e39162bdf41e4e5de9081195488f6aa7bc77ca4157115d6de70f80ab36d" gracePeriod=30
Nov 03 11:27:03.303495 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.303436    5393 generic.go:356] "Generic (PLEG): container finished" podID="a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" containerID="a2200e39162bdf41e4e5de9081195488f6aa7bc77ca4157115d6de70f80ab36d" exitCode=0
Nov 03 11:27:03.303583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.303569    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h" event={"ID":"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f","Type":"ContainerDied","Data":"a2200e39162bdf41e4e5de9081195488f6aa7bc77ca4157115d6de70f80ab36d"}
Nov 03 11:27:03.330163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.330134    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:27:03.418207 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.418171    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-metrics-certs\") pod \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") "
Nov 03 11:27:03.418422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.418242    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-kube-api-access-829zg\" (UniqueName: \"kubernetes.io/projected/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-kube-api-access-829zg\") pod \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") "
Nov 03 11:27:03.418422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.418352    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-openshift-service-ca\") pod \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") "
Nov 03 11:27:03.418422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.418417    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-maas-api-tls\") pod \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\" (UID: \"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f\") "
Nov 03 11:27:03.418732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.418707    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-openshift-service-ca" pod "a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" (UID: "a0dbc1b4-1267-40f9-90d5-f8ef3faea21f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:03.419134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.419113    5393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-openshift-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:03.420612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.420584    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-maas-api-tls" pod "a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" (UID: "a0dbc1b4-1267-40f9-90d5-f8ef3faea21f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:27:03.421082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.421061    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-metrics-certs" pod "a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" (UID: "a0dbc1b4-1267-40f9-90d5-f8ef3faea21f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:27:03.421165 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.421116    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-kube-api-access-829zg" pod "a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" (UID: "a0dbc1b4-1267-40f9-90d5-f8ef3faea21f"). InnerVolumeSpecName "kube-api-access-829zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:03.520740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.520706    5393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-metrics-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:03.520740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.520737    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829zg\" (UniqueName: \"kubernetes.io/projected/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-kube-api-access-829zg\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:03.520740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:03.520747    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f-maas-api-tls\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:04.326287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:04.326231    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h" event={"ID":"a0dbc1b4-1267-40f9-90d5-f8ef3faea21f","Type":"ContainerDied","Data":"96df586c1e4dac2f140e2e44803c20827c933944fe0a3df8325f1e3fcc3d57f5"}
Nov 03 11:27:04.326778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:04.326333    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"
Nov 03 11:27:04.326778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:04.326351    5393 scope.go:122] "RemoveContainer" containerID="a2200e39162bdf41e4e5de9081195488f6aa7bc77ca4157115d6de70f80ab36d"
Nov 03 11:27:04.345896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:04.345813    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"]
Nov 03 11:27:04.349794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:04.349770    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9f53bb-847976df7f-lgf6h"]
Nov 03 11:27:05.237521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:05.237484    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g" podUID="4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" containerName="payload-processing" containerID="cri-o://7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b" gracePeriod=25
Nov 03 11:27:06.282539 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:06.282504    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" path="/var/lib/kubelet/pods/a0dbc1b4-1267-40f9-90d5-f8ef3faea21f/volumes"
Nov 03 11:27:13.191446 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:27:13.191402    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:27:14.768016 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:14.767970    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g" podUID="4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:16.880601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:16.880566    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"]
Nov 03 11:27:16.881208 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:16.880827    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy" containerID="cri-o://0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a" gracePeriod=30
Nov 03 11:27:18.889993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:18.889945    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"]
Nov 03 11:27:18.890517 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:18.890166    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" containerID="cri-o://2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13" gracePeriod=30
Nov 03 11:27:21.210433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:21.210391    5393 scope.go:122] "RemoveContainer" containerID="4c4fcee5b8b76b9139142b0ce04893b970aa6d8747ff41e0099044c659910ba5"
Nov 03 11:27:21.215889 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:21.215860    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"]
Nov 03 11:27:22.178220 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.178190    5393 patch_prober.go:29] interesting pod/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:27:22.178361 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.178245    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:27:22.196413 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.196388    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 11:27:22.203750 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.203727    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-envoy\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.203933 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.203787    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-certs\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.203933 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.203832    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-token\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-token\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.204081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.203938    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-podinfo\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.204081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.203965    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-socket\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.204081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204001    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c280ce5-c5dd-421d-afdd-977f78ced608-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c280ce5-c5dd-421d-afdd-977f78ced608-istiod-ca-cert\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.204081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204069    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-kube-api-access-xblh9\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-kube-api-access-xblh9\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.204302 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204108    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-certs" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:22.204302 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204142    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-credential-socket\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.204302 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204198    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-data\") pod \"7c280ce5-c5dd-421d-afdd-977f78ced608\" (UID: \"7c280ce5-c5dd-421d-afdd-977f78ced608\") "
Nov 03 11:27:22.204510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204298    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-socket" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:22.204510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204442    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-credential-socket" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:22.204510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204459    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c280ce5-c5dd-421d-afdd-977f78ced608-istiod-ca-cert" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:22.204813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204639    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-data" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:22.205013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.204991    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.205107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.205019    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.205107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.205035    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.205107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.205049    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.205107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.205076    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c280ce5-c5dd-421d-afdd-977f78ced608-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.206428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.206405    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-envoy" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:22.207072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.207045    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-token" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:22.207072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.207053    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-podinfo" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:27:22.207201 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.207125    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-kube-api-access-xblh9" pod "7c280ce5-c5dd-421d-afdd-977f78ced608" (UID: "7c280ce5-c5dd-421d-afdd-977f78ced608"). InnerVolumeSpecName "kube-api-access-xblh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:22.306494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.306464    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.306494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.306489    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.306494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.306501    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xblh9\" (UniqueName: \"kubernetes.io/projected/7c280ce5-c5dd-421d-afdd-977f78ced608-kube-api-access-xblh9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.307046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.306518    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c280ce5-c5dd-421d-afdd-977f78ced608-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:22.534701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.534611    5393 generic.go:356] "Generic (PLEG): container finished" podID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerID="0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a" exitCode=0
Nov 03 11:27:22.534701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.534660    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" event={"ID":"7c280ce5-c5dd-421d-afdd-977f78ced608","Type":"ContainerDied","Data":"0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a"}
Nov 03 11:27:22.534701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.534665    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"
Nov 03 11:27:22.535000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.534713    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h" event={"ID":"7c280ce5-c5dd-421d-afdd-977f78ced608","Type":"ContainerDied","Data":"ce60fdd852c463c5093013c740ba54df3e59138d953c34a1a564a745fc150549"}
Nov 03 11:27:22.535000 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.534741    5393 scope.go:122] "RemoveContainer" containerID="0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a"
Nov 03 11:27:22.549488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.549461    5393 scope.go:122] "RemoveContainer" containerID="0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a"
Nov 03 11:27:22.549806 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:27:22.549777    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a\": container with ID starting with 0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a not found: ID does not exist" containerID="0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a"
Nov 03 11:27:22.549864 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.549819    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a"} err="failed to get container status \"0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a\": rpc error: code = NotFound desc = could not find container \"0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a\": container with ID starting with 0da84f99d45744e8903d2fb1b38a4bc86397151d07d1022e611e3aec1683182a not found: ID does not exist"
Nov 03 11:27:22.559220 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.559196    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"]
Nov 03 11:27:22.564748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:22.564723    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-47a8a4-openshift-default-54fbfbc8df-lzx6h"]
Nov 03 11:27:24.174285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.174227    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 11:27:24.233046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.233013    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" path="/var/lib/kubelet/pods/7c280ce5-c5dd-421d-afdd-977f78ced608/volumes"
Nov 03 11:27:24.234118 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234091    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istiod-ca-cert\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.234118 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234134    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-kube-api-access-bf54t\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-kube-api-access-bf54t\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.234468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234257    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-socket\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.234468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234287    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-credential-socket\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.234468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234328    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-data\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.234468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234360    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-envoy\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.234839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234487    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-podinfo\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.234839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234579    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-certs\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.234839 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.234671    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-token\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-token\") pod \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\" (UID: \"663dc3ce-c558-4fc1-ab57-217b48ae1af8\") "
Nov 03 11:27:24.235136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.235054    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-data" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:24.235496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.235468    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.235709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.235678    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-socket" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:24.235841 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.235817    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-credential-socket" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:24.235972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.235728    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-certs" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:24.236255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.236228    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istiod-ca-cert" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:24.238375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.238214    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-token" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:24.239904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.239872    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-podinfo" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:27:24.246644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.246620    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-kube-api-access-bf54t" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "kube-api-access-bf54t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:24.246644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.246633    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-envoy" pod "663dc3ce-c558-4fc1-ab57-217b48ae1af8" (UID: "663dc3ce-c558-4fc1-ab57-217b48ae1af8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:24.337344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.337304    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.337344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.337342    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.337586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.337380    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.337586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.337394    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.337586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.337408    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf54t\" (UniqueName: \"kubernetes.io/projected/663dc3ce-c558-4fc1-ab57-217b48ae1af8-kube-api-access-bf54t\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.337586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.337421    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.337586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.337433    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.337586 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.337446    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663dc3ce-c558-4fc1-ab57-217b48ae1af8-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:24.552598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.552500    5393 generic.go:356] "Generic (PLEG): container finished" podID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerID="2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13" exitCode=0
Nov 03 11:27:24.552598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.552547    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"
Nov 03 11:27:24.552598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.552584    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" event={"ID":"663dc3ce-c558-4fc1-ab57-217b48ae1af8","Type":"ContainerDied","Data":"2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13"}
Nov 03 11:27:24.552828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.552642    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8" event={"ID":"663dc3ce-c558-4fc1-ab57-217b48ae1af8","Type":"ContainerDied","Data":"6872ebc00f253772371c883cd69c68cdf25ad465fe61f83841102c3b7821c4c4"}
Nov 03 11:27:24.552828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.552664    5393 scope.go:122] "RemoveContainer" containerID="2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13"
Nov 03 11:27:24.564732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.564713    5393 scope.go:122] "RemoveContainer" containerID="2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13"
Nov 03 11:27:24.565096 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:27:24.565061    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13\": container with ID starting with 2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13 not found: ID does not exist" containerID="2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13"
Nov 03 11:27:24.565206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.565093    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13"} err="failed to get container status \"2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13\": rpc error: code = NotFound desc = could not find container \"2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13\": container with ID starting with 2f5ed640f125beaec7cbe5253317ad882790287b6ef00de1f95cdc569c859a13 not found: ID does not exist"
Nov 03 11:27:24.576688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.576659    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"]
Nov 03 11:27:24.584864 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.584839    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e78c33-openshift-default-5d4fbf8c67-js9t8"]
Nov 03 11:27:24.767293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:24.767257    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g" podUID="4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:25.507420 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.507345    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:27:25.553212 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.553176    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-plugins-config-volume\") pod \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\" (UID: \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\") "
Nov 03 11:27:25.553431 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.553291    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-kube-api-access-qpkf7\" (UniqueName: \"kubernetes.io/projected/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-kube-api-access-qpkf7\") pod \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\" (UID: \"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6\") "
Nov 03 11:27:25.553655 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.553618    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-plugins-config-volume" pod "4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" (UID: "4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:25.554104 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.554080    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:25.556204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.556177    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-kube-api-access-qpkf7" pod "4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" (UID: "4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6"). InnerVolumeSpecName "kube-api-access-qpkf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:25.565482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.564381    5393 generic.go:356] "Generic (PLEG): container finished" podID="4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" containerID="7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b" exitCode=0
Nov 03 11:27:25.565482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.564475    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g" event={"ID":"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6","Type":"ContainerDied","Data":"7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b"}
Nov 03 11:27:25.565482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.564515    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"
Nov 03 11:27:25.565482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.564534    5393 scope.go:122] "RemoveContainer" containerID="7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b"
Nov 03 11:27:25.565482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.564522    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g" event={"ID":"4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6","Type":"ContainerDied","Data":"2e4250bd0dc0b45732b2ec0e9116e47ca511ccc39e4ee95b0c4a4b899e69653b"}
Nov 03 11:27:25.582298 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.582272    5393 scope.go:122] "RemoveContainer" containerID="7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b"
Nov 03 11:27:25.582692 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:27:25.582662    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b\": container with ID starting with 7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b not found: ID does not exist" containerID="7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b"
Nov 03 11:27:25.582786 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.582705    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b"} err="failed to get container status \"7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b\": rpc error: code = NotFound desc = could not find container \"7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b\": container with ID starting with 7fa28c350206851cfa46b17a4a7f64e3c96242407e18e42bb037696d669c350b not found: ID does not exist"
Nov 03 11:27:25.597511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.597479    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"]
Nov 03 11:27:25.602794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.602765    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9f53bb-74c95f4db5-2b47g"]
Nov 03 11:27:25.655253 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:25.655220    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkf7\" (UniqueName: \"kubernetes.io/projected/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6-kube-api-access-qpkf7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:26.216797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:26.216739    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s" podUID="d3694a1b-be9a-4684-b795-ae49922bf43c" containerName="payload-pre-processing" containerID="cri-o://67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d" gracePeriod=25
Nov 03 11:27:26.233077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:26.233045    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" path="/var/lib/kubelet/pods/4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6/volumes"
Nov 03 11:27:26.233576 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:26.233563    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" path="/var/lib/kubelet/pods/663dc3ce-c558-4fc1-ab57-217b48ae1af8/volumes"
Nov 03 11:27:26.517597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:26.517500    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s" podUID="d3694a1b-be9a-4684-b795-ae49922bf43c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:36.517411 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:36.517370    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s" podUID="d3694a1b-be9a-4684-b795-ae49922bf43c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:37.944154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:37.944117    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"]
Nov 03 11:27:37.944664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:37.944415    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy" containerID="cri-o://9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89" gracePeriod=30
Nov 03 11:27:42.188753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.188715    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:27:42.189268 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.188775    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:27:42.200403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.200373    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"]
Nov 03 11:27:42.201129 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201109    5393 cpu_manager.go:414] "containerMap: removing container" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201130    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201148    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" containerName="maas-api"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201155    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" containerName="maas-api"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201174    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201183    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201204    5393 cpu_manager.go:414] "containerMap: removing container" podUID="bb4ca33d-3407-4793-97be-a091fcae22cb" containerName="revoke-keys"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201212    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4ca33d-3407-4793-97be-a091fcae22cb" containerName="revoke-keys"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201224    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" containerName="payload-processing"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201232    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" containerName="payload-processing"
Nov 03 11:27:42.201250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201249    5393 cpu_manager.go:414] "containerMap: removing container" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201257    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201267    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201274    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201284    5393 cpu_manager.go:414] "containerMap: removing container" podUID="5d297c0a-8957-4a5d-b652-4c11a35b9b0d" containerName="payload-processing"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201292    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d297c0a-8957-4a5d-b652-4c11a35b9b0d" containerName="payload-processing"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201302    5393 cpu_manager.go:414] "containerMap: removing container" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201309    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201336    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201345    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201372    5393 cpu_manager.go:414] "containerMap: removing container" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201380    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201392    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ea7380ea-3664-469f-a0d4-c947bdecb710" containerName="main"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201399    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7380ea-3664-469f-a0d4-c947bdecb710" containerName="main"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201552    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201565    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dbc1b4-1267-40f9-90d5-f8ef3faea21f" containerName="maas-api"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201576    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201588    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4ca33d-3407-4793-97be-a091fcae22cb" containerName="revoke-keys"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201598    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbf5e5c-cfe2-412c-941b-a25b9d12b9e6" containerName="payload-processing"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201611    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f308503-efc9-4f8d-9310-02616550cb70" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201621    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c280ce5-c5dd-421d-afdd-977f78ced608" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201630    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d297c0a-8957-4a5d-b652-4c11a35b9b0d" containerName="payload-processing"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201641    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a685fa6-5b4a-45bc-8877-815f154949d3" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201652    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9be7fd-4617-4579-a056-80f3bdb258cb" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201666    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="663dc3ce-c558-4fc1-ab57-217b48ae1af8" containerName="istio-proxy"
Nov 03 11:27:42.201805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.201676    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7380ea-3664-469f-a0d4-c947bdecb710" containerName="main"
Nov 03 11:27:42.207005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.206985    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.215718 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.215699    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-6012b5\"/\"default-dockercfg-njlsp\""
Nov 03 11:27:42.217093 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.217074    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-6012b5\"/\"rate-test-model-6012b5-kserve-self-signed-certs\""
Nov 03 11:27:42.217566 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.217549    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-6012b5\"/\"openshift-service-ca.crt\""
Nov 03 11:27:42.218766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.218743    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-6012b5\"/\"kube-root-ca.crt\""
Nov 03 11:27:42.239995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.239958    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"]
Nov 03 11:27:42.245621 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.245592    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-tmp-dir\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.248592 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.248567    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f572826e-378d-4514-9055-e68f3872b0ea-tls-certs\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.248878 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.248856    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-home\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.249493 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.249116    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzmn\" (UniqueName: \"kubernetes.io/projected/f572826e-378d-4514-9055-e68f3872b0ea-kube-api-access-sqzmn\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.249985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.249943    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-model-cache\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.250262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.250241    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-dshm\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.352308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.352268    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-tmp-dir\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.352514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.352361    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f572826e-378d-4514-9055-e68f3872b0ea-tls-certs\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.352514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.352398    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-home\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.352514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.352422    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzmn\" (UniqueName: \"kubernetes.io/projected/f572826e-378d-4514-9055-e68f3872b0ea-kube-api-access-sqzmn\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.352709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.352591    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-model-cache\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.352709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.352691    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-dshm\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.352845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.352757    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-tmp-dir\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.352981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.352952    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-model-cache\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.353208 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.353188    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-home\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.355603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.355552    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-dshm\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.355814 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.355739    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f572826e-378d-4514-9055-e68f3872b0ea-tls-certs\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.412669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.412632    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzmn\" (UniqueName: \"kubernetes.io/projected/f572826e-378d-4514-9055-e68f3872b0ea-kube-api-access-sqzmn\") pod \"rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") " pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.520967 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.520821    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:42.824592 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:27:42.824550    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf572826e_378d_4514_9055_e68f3872b0ea.slice/crio-dad246baf49bcd289fbba578ccf7832eac7e6f0a3001d064d9da7ee4e0fa4a4d WatchSource:0}: Error finding container dad246baf49bcd289fbba578ccf7832eac7e6f0a3001d064d9da7ee4e0fa4a4d: Status 404 returned error can't find the container with id dad246baf49bcd289fbba578ccf7832eac7e6f0a3001d064d9da7ee4e0fa4a4d
Nov 03 11:27:42.825500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:42.825477    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"]
Nov 03 11:27:43.202196 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:27:43.202153    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:27:43.208417 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.208388    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:27:43.271907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.271866    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-token\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-token\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.272121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.272097    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-envoy\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.272225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.272159    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-credential-socket\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.272225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.272209    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istiod-ca-cert\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.272340 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.272249    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-data\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.272396 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.272343    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-kube-api-access-jcjxm\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-kube-api-access-jcjxm\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.272472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.272457    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-podinfo\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.277003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.273956    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-certs\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.277003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.274343    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-socket\") pod \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\" (UID: \"ae47eae1-5e93-49ab-b252-7e5c30dd1d23\") "
Nov 03 11:27:43.277003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.275348    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-credential-socket" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:43.284587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.282465    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-envoy" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:43.284587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.284234    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-kube-api-access-jcjxm" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "kube-api-access-jcjxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:43.284827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.284594    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istiod-ca-cert" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:43.288116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.284887    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-data" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:43.288116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.286057    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-token" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:43.288116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.286255    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-certs" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:43.288116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.286846    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-socket" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:27:43.288116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.288011    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-podinfo" pod "ae47eae1-5e93-49ab-b252-7e5c30dd1d23" (UID: "ae47eae1-5e93-49ab-b252-7e5c30dd1d23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308760    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308792    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308816    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308832    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308847    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308868    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcjxm\" (UniqueName: \"kubernetes.io/projected/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-kube-api-access-jcjxm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308890    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308929    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.309206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.308945    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae47eae1-5e93-49ab-b252-7e5c30dd1d23-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:43.709269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.709235    5393 generic.go:356] "Generic (PLEG): container finished" podID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerID="9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89" exitCode=0
Nov 03 11:27:43.709494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.709314    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"
Nov 03 11:27:43.709494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.709310    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" event={"ID":"ae47eae1-5e93-49ab-b252-7e5c30dd1d23","Type":"ContainerDied","Data":"9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89"}
Nov 03 11:27:43.709494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.709473    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh" event={"ID":"ae47eae1-5e93-49ab-b252-7e5c30dd1d23","Type":"ContainerDied","Data":"1fd6b46010a6dc7a0eeef87bcaf005a0d5b1aabd4aea13ab1772bfbe4b78f0c9"}
Nov 03 11:27:43.709678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.709499    5393 scope.go:122] "RemoveContainer" containerID="9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89"
Nov 03 11:27:43.711567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.711544    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q" event={"ID":"f572826e-378d-4514-9055-e68f3872b0ea","Type":"ContainerStarted","Data":"d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b"}
Nov 03 11:27:43.711695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.711581    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q" event={"ID":"f572826e-378d-4514-9055-e68f3872b0ea","Type":"ContainerStarted","Data":"dad246baf49bcd289fbba578ccf7832eac7e6f0a3001d064d9da7ee4e0fa4a4d"}
Nov 03 11:27:43.721467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.721447    5393 scope.go:122] "RemoveContainer" containerID="8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306"
Nov 03 11:27:43.732191 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.732168    5393 scope.go:122] "RemoveContainer" containerID="9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89"
Nov 03 11:27:43.732480 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:27:43.732458    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89\": container with ID starting with 9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89 not found: ID does not exist" containerID="9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89"
Nov 03 11:27:43.732573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.732484    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89"} err="failed to get container status \"9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89\": rpc error: code = NotFound desc = could not find container \"9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89\": container with ID starting with 9235cc6b95dc4f312f3ce885c5171129bdb3b54c0016dc62643ede866db9eb89 not found: ID does not exist"
Nov 03 11:27:43.732573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.732501    5393 scope.go:122] "RemoveContainer" containerID="8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306"
Nov 03 11:27:43.732752 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:27:43.732735    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306\": container with ID starting with 8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306 not found: ID does not exist" containerID="8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306"
Nov 03 11:27:43.732811 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.732756    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306"} err="failed to get container status \"8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306\": rpc error: code = NotFound desc = could not find container \"8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306\": container with ID starting with 8a150234d5dcc252b5377392455daeb7f0290828d210b04c27c8756e7de30306 not found: ID does not exist"
Nov 03 11:27:43.746810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.746766    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q" podStartSLOduration=1.7467544990000001 podStartE2EDuration="1.746754499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:27:43.74427256 +0000 UTC m=+3507.649127001" watchObservedRunningTime="2026-11-03 11:27:43.746754499 +0000 UTC m=+3507.651608939"
Nov 03 11:27:43.768251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.768226    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"]
Nov 03 11:27:43.774995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:43.774962    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d9f2ce-openshift-default-6ffcd4dddf-jkckh"]
Nov 03 11:27:44.234267 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:44.234222    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" path="/var/lib/kubelet/pods/ae47eae1-5e93-49ab-b252-7e5c30dd1d23/volumes"
Nov 03 11:27:44.521300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:44.521219    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:46.497840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.497806    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:27:46.557617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.556353    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3694a1b-be9a-4684-b795-ae49922bf43c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3694a1b-be9a-4684-b795-ae49922bf43c-plugins-config-volume\") pod \"d3694a1b-be9a-4684-b795-ae49922bf43c\" (UID: \"d3694a1b-be9a-4684-b795-ae49922bf43c\") "
Nov 03 11:27:46.557617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.556483    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3694a1b-be9a-4684-b795-ae49922bf43c-kube-api-access-6vc89\" (UniqueName: \"kubernetes.io/projected/d3694a1b-be9a-4684-b795-ae49922bf43c-kube-api-access-6vc89\") pod \"d3694a1b-be9a-4684-b795-ae49922bf43c\" (UID: \"d3694a1b-be9a-4684-b795-ae49922bf43c\") "
Nov 03 11:27:46.557998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.557970    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3694a1b-be9a-4684-b795-ae49922bf43c-plugins-config-volume" pod "d3694a1b-be9a-4684-b795-ae49922bf43c" (UID: "d3694a1b-be9a-4684-b795-ae49922bf43c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:27:46.560358 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.560329    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3694a1b-be9a-4684-b795-ae49922bf43c-kube-api-access-6vc89" pod "d3694a1b-be9a-4684-b795-ae49922bf43c" (UID: "d3694a1b-be9a-4684-b795-ae49922bf43c"). InnerVolumeSpecName "kube-api-access-6vc89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:27:46.659058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.659022    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3694a1b-be9a-4684-b795-ae49922bf43c-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:46.659058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.659050    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vc89\" (UniqueName: \"kubernetes.io/projected/d3694a1b-be9a-4684-b795-ae49922bf43c-kube-api-access-6vc89\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:27:46.748767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.748685    5393 generic.go:356] "Generic (PLEG): container finished" podID="d3694a1b-be9a-4684-b795-ae49922bf43c" containerID="67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d" exitCode=0
Nov 03 11:27:46.748767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.748731    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s" event={"ID":"d3694a1b-be9a-4684-b795-ae49922bf43c","Type":"ContainerDied","Data":"67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d"}
Nov 03 11:27:46.749032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.748792    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s" event={"ID":"d3694a1b-be9a-4684-b795-ae49922bf43c","Type":"ContainerDied","Data":"9c321b5910f78f4b71a54f4709aaca201574c491a6dbebea8e4fa06ec38efb35"}
Nov 03 11:27:46.749032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.748816    5393 scope.go:122] "RemoveContainer" containerID="67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d"
Nov 03 11:27:46.749032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.748744    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"
Nov 03 11:27:46.760298 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.760274    5393 scope.go:122] "RemoveContainer" containerID="67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d"
Nov 03 11:27:46.760627 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:27:46.760603    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d\": container with ID starting with 67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d not found: ID does not exist" containerID="67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d"
Nov 03 11:27:46.760699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.760636    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d"} err="failed to get container status \"67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d\": rpc error: code = NotFound desc = could not find container \"67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d\": container with ID starting with 67d73e54da77c23e8ed49d8cb2b9f69ff5cd038de7e593c84711af534fbcde4d not found: ID does not exist"
Nov 03 11:27:46.775849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.775815    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"]
Nov 03 11:27:46.783866 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:46.783838    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d9f2ce-75745f59f9-rl99s"]
Nov 03 11:27:48.233315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:48.233282    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3694a1b-be9a-4684-b795-ae49922bf43c" path="/var/lib/kubelet/pods/d3694a1b-be9a-4684-b795-ae49922bf43c/volumes"
Nov 03 11:27:48.684806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:48.684769    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"]
Nov 03 11:27:48.695714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:48.695683    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"]
Nov 03 11:27:52.521131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:52.521098    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:52.535936 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:52.535897    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:52.840002 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:52.836397    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:27:53.686082 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:53.686036    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr" podUID="0c3ad30c-e2ac-4f38-a031-71f228f0f522" containerName="payload-processing" containerID="cri-o://018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761" gracePeriod=25
Nov 03 11:27:53.696584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:53.696542    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld" podUID="8519bd00-775f-484a-bb47-091b12fee81c" containerName="payload-pre-processing" containerID="cri-o://a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb" gracePeriod=25
Nov 03 11:27:56.408217 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:56.408184    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"]
Nov 03 11:27:57.543304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:57.543265    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld" podUID="8519bd00-775f-484a-bb47-091b12fee81c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:57.543784 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:57.543308    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr" podUID="0c3ad30c-e2ac-4f38-a031-71f228f0f522" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:27:58.607552 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.607325    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"]
Nov 03 11:27:58.608098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608078    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy"
Nov 03 11:27:58.608098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608098    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy"
Nov 03 11:27:58.608206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608121    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d3694a1b-be9a-4684-b795-ae49922bf43c" containerName="payload-pre-processing"
Nov 03 11:27:58.608206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608126    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3694a1b-be9a-4684-b795-ae49922bf43c" containerName="payload-pre-processing"
Nov 03 11:27:58.608206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608140    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy"
Nov 03 11:27:58.608206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608144    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy"
Nov 03 11:27:58.608391 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608270    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy"
Nov 03 11:27:58.608391 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608283    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3694a1b-be9a-4684-b795-ae49922bf43c" containerName="payload-pre-processing"
Nov 03 11:27:58.608391 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.608292    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae47eae1-5e93-49ab-b252-7e5c30dd1d23" containerName="istio-proxy"
Nov 03 11:27:58.612075 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.612048    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.622206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.622180    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"]
Nov 03 11:27:58.626255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.626204    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cc8214\"/\"default-dockercfg-8x489\""
Nov 03 11:27:58.626255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.626213    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cc8214\"/\"openshift-service-ca.crt\""
Nov 03 11:27:58.626440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.626274    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cc8214\"/\"rate-test-model-cc8214-kserve-self-signed-certs\""
Nov 03 11:27:58.626440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.626279    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cc8214\"/\"kube-root-ca.crt\""
Nov 03 11:27:58.737903 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.737848    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dncnl\" (UniqueName: \"kubernetes.io/projected/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-kube-api-access-dncnl\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.738178 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.737988    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-model-cache\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.738178 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.738021    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-home\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.738178 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.738049    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tmp-dir\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.738390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.738221    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-dshm\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.738390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.738356    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tls-certs\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.840654 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.840597    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-home\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.840970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.840949    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tmp-dir\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.841346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.841329    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-dshm\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.841621 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.841595    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tmp-dir\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.841804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.841624    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tls-certs\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.841959 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.841942    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dncnl\" (UniqueName: \"kubernetes.io/projected/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-kube-api-access-dncnl\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.842150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.842132    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-model-cache\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.842606 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.842586    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-model-cache\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.843472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.841373    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-home\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.846124 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.845747    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tls-certs\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.853793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.853766    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dncnl\" (UniqueName: \"kubernetes.io/projected/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-kube-api-access-dncnl\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.860047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.859051    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-dshm\") pod \"rate-test-model-cc8214-kserve-58668ff977-4r4qd\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") " pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:58.932249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:58.932205    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:27:59.225957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:59.225909    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"]
Nov 03 11:27:59.880021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:59.879983    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd" event={"ID":"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae","Type":"ContainerStarted","Data":"3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517"}
Nov 03 11:27:59.880494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:59.880031    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd" event={"ID":"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae","Type":"ContainerStarted","Data":"bc154c65ae2d0a57e69efa4ceeae1df021b5764f8fd43129f34a044fef8fa7b2"}
Nov 03 11:27:59.898010 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:59.897960    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd" podStartSLOduration=1.897946111 podStartE2EDuration="1.897946111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:27:59.896261546 +0000 UTC m=+3523.801115989" watchObservedRunningTime="2026-11-03 11:27:59.897946111 +0000 UTC m=+3523.802800561"
Nov 03 11:27:59.933244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:27:59.933181    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:28:01.409437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:01.409227    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6" podUID="18eda48e-713c-4144-8608-b2f7a2f56503" containerName="payload-processing" containerID="cri-o://347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c" gracePeriod=25
Nov 03 11:28:07.543418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:07.543328    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr" podUID="0c3ad30c-e2ac-4f38-a031-71f228f0f522" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:07.543418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:07.543360    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld" podUID="8519bd00-775f-484a-bb47-091b12fee81c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:08.813969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:08.807128    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6" podUID="18eda48e-713c-4144-8608-b2f7a2f56503" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:08.979960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:08.945436    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:28:09.031509 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:09.031477    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:28:09.043570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:09.043540    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:28:13.189661 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:13.189618    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:28:13.988548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:13.988520    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 11:28:14.029895 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.029864    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 11:28:14.088970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.088881    5393 generic.go:356] "Generic (PLEG): container finished" podID="8519bd00-775f-484a-bb47-091b12fee81c" containerID="a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb" exitCode=0
Nov 03 11:28:14.088970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.088955    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld" event={"ID":"8519bd00-775f-484a-bb47-091b12fee81c","Type":"ContainerDied","Data":"a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb"}
Nov 03 11:28:14.089248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.088969    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"
Nov 03 11:28:14.089248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.089061    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld" event={"ID":"8519bd00-775f-484a-bb47-091b12fee81c","Type":"ContainerDied","Data":"0fd93efd14d272a5bd32fc61b55c1a1a46df72076434e54977ebcc55152235dd"}
Nov 03 11:28:14.089248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.089092    5393 scope.go:122] "RemoveContainer" containerID="a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb"
Nov 03 11:28:14.091050 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.091021    5393 generic.go:356] "Generic (PLEG): container finished" podID="0c3ad30c-e2ac-4f38-a031-71f228f0f522" containerID="018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761" exitCode=0
Nov 03 11:28:14.091180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.091067    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"
Nov 03 11:28:14.091180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.091111    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr" event={"ID":"0c3ad30c-e2ac-4f38-a031-71f228f0f522","Type":"ContainerDied","Data":"018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761"}
Nov 03 11:28:14.091180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.091176    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr" event={"ID":"0c3ad30c-e2ac-4f38-a031-71f228f0f522","Type":"ContainerDied","Data":"4e9f42e8417bc4b3e17a0dbb21eff3e27e64b7c56eaa4de67cd70af4eaad92f1"}
Nov 03 11:28:14.100131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.100110    5393 scope.go:122] "RemoveContainer" containerID="a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb"
Nov 03 11:28:14.100458 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:14.100428    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb\": container with ID starting with a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb not found: ID does not exist" containerID="a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb"
Nov 03 11:28:14.100510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.100466    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb"} err="failed to get container status \"a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb\": rpc error: code = NotFound desc = could not find container \"a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb\": container with ID starting with a2c35d68ffcfe87c9e7aa49f9ade59c07c46057028e6eba59db1680d5e45d0fb not found: ID does not exist"
Nov 03 11:28:14.100510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.100483    5393 scope.go:122] "RemoveContainer" containerID="018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761"
Nov 03 11:28:14.110194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.110171    5393 scope.go:122] "RemoveContainer" containerID="018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761"
Nov 03 11:28:14.110452 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:14.110433    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761\": container with ID starting with 018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761 not found: ID does not exist" containerID="018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761"
Nov 03 11:28:14.110526 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.110458    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761"} err="failed to get container status \"018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761\": rpc error: code = NotFound desc = could not find container \"018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761\": container with ID starting with 018950f78edc0d0c6a0f19488e3190aed229f2c8543cee121438b6c07a6b0761 not found: ID does not exist"
Nov 03 11:28:14.165824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.165793    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8519bd00-775f-484a-bb47-091b12fee81c-kube-api-access-97dkl\" (UniqueName: \"kubernetes.io/projected/8519bd00-775f-484a-bb47-091b12fee81c-kube-api-access-97dkl\") pod \"8519bd00-775f-484a-bb47-091b12fee81c\" (UID: \"8519bd00-775f-484a-bb47-091b12fee81c\") "
Nov 03 11:28:14.165824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.165834    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8519bd00-775f-484a-bb47-091b12fee81c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519bd00-775f-484a-bb47-091b12fee81c-plugins-config-volume\") pod \"8519bd00-775f-484a-bb47-091b12fee81c\" (UID: \"8519bd00-775f-484a-bb47-091b12fee81c\") "
Nov 03 11:28:14.166127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.165908    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3ad30c-e2ac-4f38-a031-71f228f0f522-kube-api-access-k7bb4\" (UniqueName: \"kubernetes.io/projected/0c3ad30c-e2ac-4f38-a031-71f228f0f522-kube-api-access-k7bb4\") pod \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\" (UID: \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\") "
Nov 03 11:28:14.166127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.166006    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c3ad30c-e2ac-4f38-a031-71f228f0f522-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3ad30c-e2ac-4f38-a031-71f228f0f522-plugins-config-volume\") pod \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\" (UID: \"0c3ad30c-e2ac-4f38-a031-71f228f0f522\") "
Nov 03 11:28:14.166326 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.166302    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8519bd00-775f-484a-bb47-091b12fee81c-plugins-config-volume" pod "8519bd00-775f-484a-bb47-091b12fee81c" (UID: "8519bd00-775f-484a-bb47-091b12fee81c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:14.166326 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.166309    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3ad30c-e2ac-4f38-a031-71f228f0f522-plugins-config-volume" pod "0c3ad30c-e2ac-4f38-a031-71f228f0f522" (UID: "0c3ad30c-e2ac-4f38-a031-71f228f0f522"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:14.166609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.166592    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c3ad30c-e2ac-4f38-a031-71f228f0f522-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:14.166652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.166612    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8519bd00-775f-484a-bb47-091b12fee81c-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:14.168186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.168167    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8519bd00-775f-484a-bb47-091b12fee81c-kube-api-access-97dkl" pod "8519bd00-775f-484a-bb47-091b12fee81c" (UID: "8519bd00-775f-484a-bb47-091b12fee81c"). InnerVolumeSpecName "kube-api-access-97dkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:14.168265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.168225    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3ad30c-e2ac-4f38-a031-71f228f0f522-kube-api-access-k7bb4" pod "0c3ad30c-e2ac-4f38-a031-71f228f0f522" (UID: "0c3ad30c-e2ac-4f38-a031-71f228f0f522"). InnerVolumeSpecName "kube-api-access-k7bb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:14.267689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.267657    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97dkl\" (UniqueName: \"kubernetes.io/projected/8519bd00-775f-484a-bb47-091b12fee81c-kube-api-access-97dkl\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:14.267689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.267682    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bb4\" (UniqueName: \"kubernetes.io/projected/0c3ad30c-e2ac-4f38-a031-71f228f0f522-kube-api-access-k7bb4\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:14.408286 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.408252    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"]
Nov 03 11:28:14.415703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.415668    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-47a8a4-778f5555cf-zftld"]
Nov 03 11:28:14.424679 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.424648    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"]
Nov 03 11:28:14.428386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:14.428365    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-47a8a4-774b5b78cf-465dr"]
Nov 03 11:28:16.249846 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:16.249780    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3ad30c-e2ac-4f38-a031-71f228f0f522" path="/var/lib/kubelet/pods/0c3ad30c-e2ac-4f38-a031-71f228f0f522/volumes"
Nov 03 11:28:16.251395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:16.251327    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8519bd00-775f-484a-bb47-091b12fee81c" path="/var/lib/kubelet/pods/8519bd00-775f-484a-bb47-091b12fee81c/volumes"
Nov 03 11:28:18.806365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:18.806327    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6" podUID="18eda48e-713c-4144-8608-b2f7a2f56503" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:21.291878 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.291844    5393 scope.go:122] "RemoveContainer" containerID="faf88e517aa6c7af871e0cf7ace0817e08152da08b83df45b0133c8a9618fa3f"
Nov 03 11:28:21.305113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.305085    5393 scope.go:122] "RemoveContainer" containerID="0f4f0e8250b3dbabad9cbc47857acf47799aafe5e308bf06e54479f36d933be4"
Nov 03 11:28:21.322703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.322675    5393 scope.go:122] "RemoveContainer" containerID="b02c5a298480496f2bcffa0fb7c31d67ef899b37447ef435e42031fe29810de6"
Nov 03 11:28:21.723309 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.723283    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:28:21.819098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.819059    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18eda48e-713c-4144-8608-b2f7a2f56503-kube-api-access-fs4c2\" (UniqueName: \"kubernetes.io/projected/18eda48e-713c-4144-8608-b2f7a2f56503-kube-api-access-fs4c2\") pod \"18eda48e-713c-4144-8608-b2f7a2f56503\" (UID: \"18eda48e-713c-4144-8608-b2f7a2f56503\") "
Nov 03 11:28:21.819319 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.819142    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18eda48e-713c-4144-8608-b2f7a2f56503-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eda48e-713c-4144-8608-b2f7a2f56503-plugins-config-volume\") pod \"18eda48e-713c-4144-8608-b2f7a2f56503\" (UID: \"18eda48e-713c-4144-8608-b2f7a2f56503\") "
Nov 03 11:28:21.820444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.820415    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18eda48e-713c-4144-8608-b2f7a2f56503-plugins-config-volume" pod "18eda48e-713c-4144-8608-b2f7a2f56503" (UID: "18eda48e-713c-4144-8608-b2f7a2f56503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:21.824080 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.822384    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18eda48e-713c-4144-8608-b2f7a2f56503-kube-api-access-fs4c2" pod "18eda48e-713c-4144-8608-b2f7a2f56503" (UID: "18eda48e-713c-4144-8608-b2f7a2f56503"). InnerVolumeSpecName "kube-api-access-fs4c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:21.921227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.921193    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18eda48e-713c-4144-8608-b2f7a2f56503-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:21.921227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:21.921224    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs4c2\" (UniqueName: \"kubernetes.io/projected/18eda48e-713c-4144-8608-b2f7a2f56503-kube-api-access-fs4c2\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:22.276598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.275478    5393 generic.go:356] "Generic (PLEG): container finished" podID="18eda48e-713c-4144-8608-b2f7a2f56503" containerID="347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c" exitCode=0
Nov 03 11:28:22.278106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.277034    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"
Nov 03 11:28:22.315156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.302852    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6" event={"ID":"18eda48e-713c-4144-8608-b2f7a2f56503","Type":"ContainerDied","Data":"347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c"}
Nov 03 11:28:22.315156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.302966    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6" event={"ID":"18eda48e-713c-4144-8608-b2f7a2f56503","Type":"ContainerDied","Data":"ba50dc607e02a5892cd4b23af7a3c0ce253c4faf8f12c148fafdafbc9950e0c4"}
Nov 03 11:28:22.315156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.303032    5393 scope.go:122] "RemoveContainer" containerID="347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c"
Nov 03 11:28:22.330539 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.330510    5393 scope.go:122] "RemoveContainer" containerID="347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c"
Nov 03 11:28:22.330882 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:22.330859    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c\": container with ID starting with 347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c not found: ID does not exist" containerID="347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c"
Nov 03 11:28:22.330979 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.330891    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c"} err="failed to get container status \"347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c\": rpc error: code = NotFound desc = could not find container \"347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c\": container with ID starting with 347925e64ff28818d67c35b3f9b0ab3c02c8657ec040004a34e17cabd085f26c not found: ID does not exist"
Nov 03 11:28:22.338219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.338195    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"]
Nov 03 11:28:22.342293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:22.342266    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cbeebd-66677c6dd9-9vbb6"]
Nov 03 11:28:24.232713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:24.232681    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18eda48e-713c-4144-8608-b2f7a2f56503" path="/var/lib/kubelet/pods/18eda48e-713c-4144-8608-b2f7a2f56503/volumes"
Nov 03 11:28:26.827051 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:26.826910    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"]
Nov 03 11:28:26.834423 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:26.834394    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"]
Nov 03 11:28:31.827808 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:31.827771    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg" podUID="d923e57f-9487-4585-8fb9-c104d09b00aa" containerName="payload-processing" containerID="cri-o://1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154" gracePeriod=25
Nov 03 11:28:31.836113 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:31.836080    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm" podUID="d78d286e-868c-41b7-bafb-a016941cf7ab" containerName="payload-pre-processing" containerID="cri-o://4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673" gracePeriod=25
Nov 03 11:28:37.722389 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:37.722352    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm" podUID="d78d286e-868c-41b7-bafb-a016941cf7ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:38.740172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:38.740137    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg" podUID="d923e57f-9487-4585-8fb9-c104d09b00aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:39.397136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:39.397096    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"]
Nov 03 11:28:39.397389 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:39.397360    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy" containerID="cri-o://1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4" gracePeriod=30
Nov 03 11:28:43.192740 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:43.192704    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:28:44.475906 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:44.475863    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33ed6e4_f8e4_4f29_8187_458a85720103.slice/crio-conmon-1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:28:44.697385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.697309    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 11:28:44.797428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797388    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-socket\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.797644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797496    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e33ed6e4-f8e4-4f29-8187-458a85720103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33ed6e4-f8e4-4f29-8187-458a85720103-istiod-ca-cert\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.797644 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797566    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-certs\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.797781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797678    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-socket" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:44.797781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797709    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-envoy\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.797962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797817    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-podinfo\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.797962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797848    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-certs" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:44.797962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797853    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-token\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-token\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.798136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797968    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-data\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.798136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.797987    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33ed6e4-f8e4-4f29-8187-458a85720103-istiod-ca-cert" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:44.798136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.798000    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-kube-api-access-gxqbw\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-kube-api-access-gxqbw\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.798136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.798101    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-credential-socket\") pod \"e33ed6e4-f8e4-4f29-8187-458a85720103\" (UID: \"e33ed6e4-f8e4-4f29-8187-458a85720103\") "
Nov 03 11:28:44.798354 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.798259    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-data" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:44.798570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.798542    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-credential-socket" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:44.798964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.798906    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.798964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.798969    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.798964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.798980    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.799221 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.798993    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33ed6e4-f8e4-4f29-8187-458a85720103-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.799221 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.799004    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.800424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.800402    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-token" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:44.800548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.800522    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-envoy" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:44.800659 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.800586    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-podinfo" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:28:44.801701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.801680    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-kube-api-access-gxqbw" pod "e33ed6e4-f8e4-4f29-8187-458a85720103" (UID: "e33ed6e4-f8e4-4f29-8187-458a85720103"). InnerVolumeSpecName "kube-api-access-gxqbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:44.900973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.900908    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.900973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.900972    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.900973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.900982    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.901251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.900991    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqbw\" (UniqueName: \"kubernetes.io/projected/e33ed6e4-f8e4-4f29-8187-458a85720103-kube-api-access-gxqbw\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:44.971110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.971009    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"]
Nov 03 11:28:44.985422 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:44.985390    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"]
Nov 03 11:28:45.465089 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.465050    5393 generic.go:356] "Generic (PLEG): container finished" podID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerID="1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4" exitCode=0
Nov 03 11:28:45.465303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.465125    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"
Nov 03 11:28:45.465303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.465136    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" event={"ID":"e33ed6e4-f8e4-4f29-8187-458a85720103","Type":"ContainerDied","Data":"1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4"}
Nov 03 11:28:45.465303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.465201    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp" event={"ID":"e33ed6e4-f8e4-4f29-8187-458a85720103","Type":"ContainerDied","Data":"ea18298e98dbc986a49e4165364dd9d34380ba93481412be90a3b0e64c7183b4"}
Nov 03 11:28:45.465303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.465226    5393 scope.go:122] "RemoveContainer" containerID="1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4"
Nov 03 11:28:45.478188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.478162    5393 scope.go:122] "RemoveContainer" containerID="e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8"
Nov 03 11:28:45.488349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.488323    5393 scope.go:122] "RemoveContainer" containerID="1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4"
Nov 03 11:28:45.488706 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:45.488684    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4\": container with ID starting with 1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4 not found: ID does not exist" containerID="1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4"
Nov 03 11:28:45.488802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.488718    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4"} err="failed to get container status \"1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4\": rpc error: code = NotFound desc = could not find container \"1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4\": container with ID starting with 1c2fe2cacad293ea0e6ce4badf6d622d3e77c4749339d971b90081237d6783f4 not found: ID does not exist"
Nov 03 11:28:45.488802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.488745    5393 scope.go:122] "RemoveContainer" containerID="e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8"
Nov 03 11:28:45.489095 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:45.489070    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8\": container with ID starting with e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8 not found: ID does not exist" containerID="e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8"
Nov 03 11:28:45.489159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.489103    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8"} err="failed to get container status \"e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8\": rpc error: code = NotFound desc = could not find container \"e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8\": container with ID starting with e4fbc415bff2f1e22f444007c24ee05a59f8d21bd43a2a88722f4523521fede8 not found: ID does not exist"
Nov 03 11:28:45.497013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.496987    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"]
Nov 03 11:28:45.499389 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:45.499369    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e11de4-openshift-default-c8b8f9f65-p6fhp"]
Nov 03 11:28:46.232288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:46.232253    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" path="/var/lib/kubelet/pods/e33ed6e4-f8e4-4f29-8187-458a85720103/volumes"
Nov 03 11:28:47.722299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:47.722253    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm" podUID="d78d286e-868c-41b7-bafb-a016941cf7ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:48.739998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:48.739957    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg" podUID="d923e57f-9487-4585-8fb9-c104d09b00aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:28:49.766759 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:49.766718    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"]
Nov 03 11:28:49.767292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:49.767024    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" containerID="cri-o://1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d" gracePeriod=30
Nov 03 11:28:52.155871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.155840    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:28:52.167055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.166312    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d923e57f-9487-4585-8fb9-c104d09b00aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923e57f-9487-4585-8fb9-c104d09b00aa-plugins-config-volume\") pod \"d923e57f-9487-4585-8fb9-c104d09b00aa\" (UID: \"d923e57f-9487-4585-8fb9-c104d09b00aa\") "
Nov 03 11:28:52.167055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.166454    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d923e57f-9487-4585-8fb9-c104d09b00aa-kube-api-access-dlqgn\" (UniqueName: \"kubernetes.io/projected/d923e57f-9487-4585-8fb9-c104d09b00aa-kube-api-access-dlqgn\") pod \"d923e57f-9487-4585-8fb9-c104d09b00aa\" (UID: \"d923e57f-9487-4585-8fb9-c104d09b00aa\") "
Nov 03 11:28:52.167055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.167011    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d923e57f-9487-4585-8fb9-c104d09b00aa-plugins-config-volume" pod "d923e57f-9487-4585-8fb9-c104d09b00aa" (UID: "d923e57f-9487-4585-8fb9-c104d09b00aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:52.168080 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.168054    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d923e57f-9487-4585-8fb9-c104d09b00aa-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:52.170729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.170700    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d923e57f-9487-4585-8fb9-c104d09b00aa-kube-api-access-dlqgn" pod "d923e57f-9487-4585-8fb9-c104d09b00aa" (UID: "d923e57f-9487-4585-8fb9-c104d09b00aa"). InnerVolumeSpecName "kube-api-access-dlqgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:52.204771 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.204735    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:28:52.268737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.268706    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78d286e-868c-41b7-bafb-a016941cf7ab-kube-api-access-mxct2\" (UniqueName: \"kubernetes.io/projected/d78d286e-868c-41b7-bafb-a016941cf7ab-kube-api-access-mxct2\") pod \"d78d286e-868c-41b7-bafb-a016941cf7ab\" (UID: \"d78d286e-868c-41b7-bafb-a016941cf7ab\") "
Nov 03 11:28:52.268956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.268815    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78d286e-868c-41b7-bafb-a016941cf7ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78d286e-868c-41b7-bafb-a016941cf7ab-plugins-config-volume\") pod \"d78d286e-868c-41b7-bafb-a016941cf7ab\" (UID: \"d78d286e-868c-41b7-bafb-a016941cf7ab\") "
Nov 03 11:28:52.269258 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.269219    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78d286e-868c-41b7-bafb-a016941cf7ab-plugins-config-volume" pod "d78d286e-868c-41b7-bafb-a016941cf7ab" (UID: "d78d286e-868c-41b7-bafb-a016941cf7ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:52.269763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.269731    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d78d286e-868c-41b7-bafb-a016941cf7ab-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:52.269860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.269768    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlqgn\" (UniqueName: \"kubernetes.io/projected/d923e57f-9487-4585-8fb9-c104d09b00aa-kube-api-access-dlqgn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:52.271483 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.271457    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78d286e-868c-41b7-bafb-a016941cf7ab-kube-api-access-mxct2" pod "d78d286e-868c-41b7-bafb-a016941cf7ab" (UID: "d78d286e-868c-41b7-bafb-a016941cf7ab"). InnerVolumeSpecName "kube-api-access-mxct2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:52.371674 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.371642    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxct2\" (UniqueName: \"kubernetes.io/projected/d78d286e-868c-41b7-bafb-a016941cf7ab-kube-api-access-mxct2\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:52.543861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.543829    5393 generic.go:356] "Generic (PLEG): container finished" podID="d78d286e-868c-41b7-bafb-a016941cf7ab" containerID="4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673" exitCode=0
Nov 03 11:28:52.544101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.543880    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"
Nov 03 11:28:52.544101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.543933    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm" event={"ID":"d78d286e-868c-41b7-bafb-a016941cf7ab","Type":"ContainerDied","Data":"4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673"}
Nov 03 11:28:52.544101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.543999    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm" event={"ID":"d78d286e-868c-41b7-bafb-a016941cf7ab","Type":"ContainerDied","Data":"cc2a6fbdff38c948129fad9c59c63e52e9f3ef01f91840fd02b6b6fe764bafac"}
Nov 03 11:28:52.544101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.544032    5393 scope.go:122] "RemoveContainer" containerID="4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673"
Nov 03 11:28:52.545466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.545442    5393 generic.go:356] "Generic (PLEG): container finished" podID="d923e57f-9487-4585-8fb9-c104d09b00aa" containerID="1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154" exitCode=0
Nov 03 11:28:52.545567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.545538    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg" event={"ID":"d923e57f-9487-4585-8fb9-c104d09b00aa","Type":"ContainerDied","Data":"1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154"}
Nov 03 11:28:52.545631 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.545548    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"
Nov 03 11:28:52.545698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.545583    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg" event={"ID":"d923e57f-9487-4585-8fb9-c104d09b00aa","Type":"ContainerDied","Data":"6f0fd5a3d65f41edcb2fc956502c9b95a90162884182b06585fb51db250b0c05"}
Nov 03 11:28:52.556013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.555997    5393 scope.go:122] "RemoveContainer" containerID="4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673"
Nov 03 11:28:52.556351 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:52.556320    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673\": container with ID starting with 4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673 not found: ID does not exist" containerID="4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673"
Nov 03 11:28:52.556452 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.556363    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673"} err="failed to get container status \"4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673\": rpc error: code = NotFound desc = could not find container \"4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673\": container with ID starting with 4c9946d80e5bc69c1242aef80eca9fa8a0754a2983fd1e8ea90aa0c3bcbcd673 not found: ID does not exist"
Nov 03 11:28:52.556452 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.556386    5393 scope.go:122] "RemoveContainer" containerID="1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154"
Nov 03 11:28:52.569801 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.569781    5393 scope.go:122] "RemoveContainer" containerID="1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154"
Nov 03 11:28:52.572139 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:52.572045    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154\": container with ID starting with 1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154 not found: ID does not exist" containerID="1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154"
Nov 03 11:28:52.572139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.572088    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154"} err="failed to get container status \"1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154\": rpc error: code = NotFound desc = could not find container \"1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154\": container with ID starting with 1b806f6379c4ce5c5ca9bfea35cc902678dbea5ee7e545a453206bbe45935154 not found: ID does not exist"
Nov 03 11:28:52.572139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.572135    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"]
Nov 03 11:28:52.577581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.577553    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b76c62-65dc767945-zw7rg"]
Nov 03 11:28:52.585613 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.585587    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"]
Nov 03 11:28:52.590189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.590164    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b76c62-69b644f695-msztm"]
Nov 03 11:28:52.948320 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.948274    5393 patch_prober.go:29] interesting pod/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:28:52.948491 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:52.948344    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:28:54.233123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:54.233084    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78d286e-868c-41b7-bafb-a016941cf7ab" path="/var/lib/kubelet/pods/d78d286e-868c-41b7-bafb-a016941cf7ab/volumes"
Nov 03 11:28:54.233769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:54.233734    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d923e57f-9487-4585-8fb9-c104d09b00aa" path="/var/lib/kubelet/pods/d923e57f-9487-4585-8fb9-c104d09b00aa/volumes"
Nov 03 11:28:55.066996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.066972    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:28:55.113300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.113229    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-kube-api-access-drh62\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-kube-api-access-drh62\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.114109 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.114072    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-podinfo\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.114244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.114125    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-data\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.114321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.114247    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-certs\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.114382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.114320    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-socket\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.114563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.114417    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-token\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.114563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.114446    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istiod-ca-cert\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.114563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.114473    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-envoy\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.114563 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.114513    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-credential-socket\") pod \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\" (UID: \"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9\") "
Nov 03 11:28:55.115498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.115467    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-credential-socket" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:55.116097 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.115936    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-socket" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:55.116387 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.116228    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-data" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:55.116777 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.116566    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-certs" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:55.117265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.117024    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istiod-ca-cert" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:28:55.125541 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.125187    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-kube-api-access-drh62" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "kube-api-access-drh62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:55.133564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.133526    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-token" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:28:55.141463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.141024    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-envoy" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:28:55.142306 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.142272    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-podinfo" pod "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" (UID: "f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:28:55.216219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216179    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.216219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216213    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.216219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216226    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.216466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216235    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.216466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216243    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.216466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216252    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.216466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216262    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drh62\" (UniqueName: \"kubernetes.io/projected/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-kube-api-access-drh62\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.216466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216272    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.216466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.216280    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:28:55.570874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.570839    5393 generic.go:356] "Generic (PLEG): container finished" podID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerID="1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d" exitCode=0
Nov 03 11:28:55.571406 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.570895    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"
Nov 03 11:28:55.571406 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.570943    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" event={"ID":"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9","Type":"ContainerDied","Data":"1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d"}
Nov 03 11:28:55.571406 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.571012    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb" event={"ID":"f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9","Type":"ContainerDied","Data":"a4dbcb09735593bc7e3bd30cf3bf3842139e8a2402030b166d8a38ab24ddf300"}
Nov 03 11:28:55.571406 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.571047    5393 scope.go:122] "RemoveContainer" containerID="1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d"
Nov 03 11:28:55.582296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.582273    5393 scope.go:122] "RemoveContainer" containerID="4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b"
Nov 03 11:28:55.593064 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.593041    5393 scope.go:122] "RemoveContainer" containerID="1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d"
Nov 03 11:28:55.593357 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:55.593334    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d\": container with ID starting with 1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d not found: ID does not exist" containerID="1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d"
Nov 03 11:28:55.593482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.593368    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d"} err="failed to get container status \"1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d\": rpc error: code = NotFound desc = could not find container \"1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d\": container with ID starting with 1308ce8137fca6856a2bf42e2cfe663d5b2549b39c44115d6c36f4d4cb69279d not found: ID does not exist"
Nov 03 11:28:55.593482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.593391    5393 scope.go:122] "RemoveContainer" containerID="4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b"
Nov 03 11:28:55.593801 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:28:55.593773    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b\": container with ID starting with 4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b not found: ID does not exist" containerID="4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b"
Nov 03 11:28:55.593958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.593802    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b"} err="failed to get container status \"4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b\": rpc error: code = NotFound desc = could not find container \"4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b\": container with ID starting with 4bb390c35c469cab40759ef65af728a827e45210f386fb76e0a45db45984302b not found: ID does not exist"
Nov 03 11:28:55.596355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.596331    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"]
Nov 03 11:28:55.600178 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:55.600158    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b76c62-openshift-default-7f44764669-7frxb"]
Nov 03 11:28:56.243784 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:56.243743    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" path="/var/lib/kubelet/pods/f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9/volumes"
Nov 03 11:28:58.549322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.549290    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"]
Nov 03 11:28:58.550083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550048    5393 cpu_manager.go:414] "containerMap: removing container" podUID="8519bd00-775f-484a-bb47-091b12fee81c" containerName="payload-pre-processing"
Nov 03 11:28:58.550083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550076    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8519bd00-775f-484a-bb47-091b12fee81c" containerName="payload-pre-processing"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550091    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550097    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550106    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550111    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550118    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d78d286e-868c-41b7-bafb-a016941cf7ab" containerName="payload-pre-processing"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550123    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78d286e-868c-41b7-bafb-a016941cf7ab" containerName="payload-pre-processing"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550144    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550152    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550164    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550169    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550175    5393 cpu_manager.go:414] "containerMap: removing container" podUID="18eda48e-713c-4144-8608-b2f7a2f56503" containerName="payload-processing"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550179    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18eda48e-713c-4144-8608-b2f7a2f56503" containerName="payload-processing"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550203    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d923e57f-9487-4585-8fb9-c104d09b00aa" containerName="payload-processing"
Nov 03 11:28:58.550222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550207    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d923e57f-9487-4585-8fb9-c104d09b00aa" containerName="payload-processing"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550235    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0c3ad30c-e2ac-4f38-a031-71f228f0f522" containerName="payload-processing"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550241    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3ad30c-e2ac-4f38-a031-71f228f0f522" containerName="payload-processing"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550373    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3ad30c-e2ac-4f38-a031-71f228f0f522" containerName="payload-processing"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550390    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8519bd00-775f-484a-bb47-091b12fee81c" containerName="payload-pre-processing"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550401    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550411    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550419    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78d286e-868c-41b7-bafb-a016941cf7ab" containerName="payload-pre-processing"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550432    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33ed6e4-f8e4-4f29-8187-458a85720103" containerName="istio-proxy"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550443    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="18eda48e-713c-4144-8608-b2f7a2f56503" containerName="payload-processing"
Nov 03 11:28:58.550845 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.550455    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d923e57f-9487-4585-8fb9-c104d09b00aa" containerName="payload-processing"
Nov 03 11:28:58.556280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.556252    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.563136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.563106    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"]
Nov 03 11:28:58.650907 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.650876    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:28:58.651092 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.650979    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:28:58.665758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.665721    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.665940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.665861    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8dft\" (UniqueName: \"kubernetes.io/projected/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-kube-api-access-r8dft\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.666013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.666001    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.767311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.767265    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.767558 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.767425    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8dft\" (UniqueName: \"kubernetes.io/projected/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-kube-api-access-r8dft\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.767640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.767563    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.768308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.768280    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.770124 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.770099    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.775965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.775941    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8dft\" (UniqueName: \"kubernetes.io/projected/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-kube-api-access-r8dft\") pod \"maas-api-revoke-sub-a11d4351-8nzvz\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:58.881191 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:58.881151    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:28:59.167197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:59.167173    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"]
Nov 03 11:28:59.169987 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:28:59.169934    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf942b2d_bf1e_4300_8fe7_ad6f602d7ce3.slice/crio-94a14ac45d00254059a7cab767d3b3f6134261fd2ce47f5a01cd9363fa2b962a WatchSource:0}: Error finding container 94a14ac45d00254059a7cab767d3b3f6134261fd2ce47f5a01cd9363fa2b962a: Status 404 returned error can't find the container with id 94a14ac45d00254059a7cab767d3b3f6134261fd2ce47f5a01cd9363fa2b962a
Nov 03 11:28:59.617491 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:59.617443    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz" event={"ID":"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3","Type":"ContainerStarted","Data":"71b583eb268847a66e0e1ef9e55f51072a7ac96cc188657c600dd2652dc5bfaa"}
Nov 03 11:28:59.617904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:59.617507    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz" event={"ID":"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3","Type":"ContainerStarted","Data":"94a14ac45d00254059a7cab767d3b3f6134261fd2ce47f5a01cd9363fa2b962a"}
Nov 03 11:28:59.642518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:59.642466    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz" podStartSLOduration=1.642451723 podStartE2EDuration="1.642451723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:28:59.638464825 +0000 UTC m=+3583.543319265" watchObservedRunningTime="2026-11-03 11:28:59.642451723 +0000 UTC m=+3583.547306163"
Nov 03 11:28:59.997935 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:28:59.997861    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" containerID="cri-o://f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397" gracePeriod=45
Nov 03 11:29:00.010482 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.010445    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main" containerID="cri-o://97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d" gracePeriod=45
Nov 03 11:29:00.365791 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.365764    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 11:29:00.369284 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.369260    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:29:00.443735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.443702    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"]
Nov 03 11:29:00.460072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.460032    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"]
Nov 03 11:29:00.496804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.496763    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a119d8-1185-4b77-b7d4-9f445928fec8-tls-certs\" (UniqueName: \"kubernetes.io/secret/86a119d8-1185-4b77-b7d4-9f445928fec8-tls-certs\") pod \"86a119d8-1185-4b77-b7d4-9f445928fec8\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") "
Nov 03 11:29:00.497018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.496829    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-home\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-home\") pod \"86a119d8-1185-4b77-b7d4-9f445928fec8\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") "
Nov 03 11:29:00.497018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.496874    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64742526-7053-46b4-afd4-5043ebbda083-tls-certs\" (UniqueName: \"kubernetes.io/secret/64742526-7053-46b4-afd4-5043ebbda083-tls-certs\") pod \"64742526-7053-46b4-afd4-5043ebbda083\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") "
Nov 03 11:29:00.497018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497013    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a119d8-1185-4b77-b7d4-9f445928fec8-kube-api-access-7htzk\" (UniqueName: \"kubernetes.io/projected/86a119d8-1185-4b77-b7d4-9f445928fec8-kube-api-access-7htzk\") pod \"86a119d8-1185-4b77-b7d4-9f445928fec8\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") "
Nov 03 11:29:00.497226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497054    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-dshm\") pod \"86a119d8-1185-4b77-b7d4-9f445928fec8\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") "
Nov 03 11:29:00.497226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497087    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-model-cache\") pod \"64742526-7053-46b4-afd4-5043ebbda083\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") "
Nov 03 11:29:00.497226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497136    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-home\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-home\") pod \"64742526-7053-46b4-afd4-5043ebbda083\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") "
Nov 03 11:29:00.497226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497163    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-home" pod "86a119d8-1185-4b77-b7d4-9f445928fec8" (UID: "86a119d8-1185-4b77-b7d4-9f445928fec8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.497226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497191    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-tmp-dir\") pod \"64742526-7053-46b4-afd4-5043ebbda083\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") "
Nov 03 11:29:00.497515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497229    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-model-cache\") pod \"86a119d8-1185-4b77-b7d4-9f445928fec8\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") "
Nov 03 11:29:00.497515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497260    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-tmp-dir\") pod \"86a119d8-1185-4b77-b7d4-9f445928fec8\" (UID: \"86a119d8-1185-4b77-b7d4-9f445928fec8\") "
Nov 03 11:29:00.497515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497490    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-tmp-dir" pod "86a119d8-1185-4b77-b7d4-9f445928fec8" (UID: "86a119d8-1185-4b77-b7d4-9f445928fec8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.497701 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497677    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-tmp-dir" pod "64742526-7053-46b4-afd4-5043ebbda083" (UID: "64742526-7053-46b4-afd4-5043ebbda083"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.497807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497696    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-home" pod "64742526-7053-46b4-afd4-5043ebbda083" (UID: "64742526-7053-46b4-afd4-5043ebbda083"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.497876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497804    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-dshm\") pod \"64742526-7053-46b4-afd4-5043ebbda083\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") "
Nov 03 11:29:00.497876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497869    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64742526-7053-46b4-afd4-5043ebbda083-kube-api-access-k88w8\" (UniqueName: \"kubernetes.io/projected/64742526-7053-46b4-afd4-5043ebbda083-kube-api-access-k88w8\") pod \"64742526-7053-46b4-afd4-5043ebbda083\" (UID: \"64742526-7053-46b4-afd4-5043ebbda083\") "
Nov 03 11:29:00.498024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.497932    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-model-cache" pod "64742526-7053-46b4-afd4-5043ebbda083" (UID: "64742526-7053-46b4-afd4-5043ebbda083"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.498336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.498295    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-model-cache" pod "86a119d8-1185-4b77-b7d4-9f445928fec8" (UID: "86a119d8-1185-4b77-b7d4-9f445928fec8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.499048 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.499020    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.499171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.499055    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.499171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.499069    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.499171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.499082    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.499171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.499096    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.499171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.499106    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.502755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.502728    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a119d8-1185-4b77-b7d4-9f445928fec8-kube-api-access-7htzk" pod "86a119d8-1185-4b77-b7d4-9f445928fec8" (UID: "86a119d8-1185-4b77-b7d4-9f445928fec8"). InnerVolumeSpecName "kube-api-access-7htzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:00.507283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.507241    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-dshm" pod "64742526-7053-46b4-afd4-5043ebbda083" (UID: "64742526-7053-46b4-afd4-5043ebbda083"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.507283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.507244    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64742526-7053-46b4-afd4-5043ebbda083-tls-certs" pod "64742526-7053-46b4-afd4-5043ebbda083" (UID: "64742526-7053-46b4-afd4-5043ebbda083"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:00.507283 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.507261    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-dshm" pod "86a119d8-1185-4b77-b7d4-9f445928fec8" (UID: "86a119d8-1185-4b77-b7d4-9f445928fec8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:00.507445 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.507424    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a119d8-1185-4b77-b7d4-9f445928fec8-tls-certs" pod "86a119d8-1185-4b77-b7d4-9f445928fec8" (UID: "86a119d8-1185-4b77-b7d4-9f445928fec8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:00.508660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.508637    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64742526-7053-46b4-afd4-5043ebbda083-kube-api-access-k88w8" pod "64742526-7053-46b4-afd4-5043ebbda083" (UID: "64742526-7053-46b4-afd4-5043ebbda083"). InnerVolumeSpecName "kube-api-access-k88w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:00.600351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.600321    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7htzk\" (UniqueName: \"kubernetes.io/projected/86a119d8-1185-4b77-b7d4-9f445928fec8-kube-api-access-7htzk\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.600351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.600351    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a119d8-1185-4b77-b7d4-9f445928fec8-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.600562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.600361    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64742526-7053-46b4-afd4-5043ebbda083-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.600562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.600369    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88w8\" (UniqueName: \"kubernetes.io/projected/64742526-7053-46b4-afd4-5043ebbda083-kube-api-access-k88w8\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.600562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.600378    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a119d8-1185-4b77-b7d4-9f445928fec8-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.600562 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.600387    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64742526-7053-46b4-afd4-5043ebbda083-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:00.630461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.630432    5393 generic.go:356] "Generic (PLEG): container finished" podID="64742526-7053-46b4-afd4-5043ebbda083" containerID="97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d" exitCode=0
Nov 03 11:29:00.630929 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.630481    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"
Nov 03 11:29:00.630929 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.630522    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" event={"ID":"64742526-7053-46b4-afd4-5043ebbda083","Type":"ContainerDied","Data":"97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d"}
Nov 03 11:29:00.630929 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.630567    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb" event={"ID":"64742526-7053-46b4-afd4-5043ebbda083","Type":"ContainerDied","Data":"f75818b558e60ee9a2578f1c1c06c300038e74a0fbb11fe4c3a3d1680d5dbbae"}
Nov 03 11:29:00.630929 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.630589    5393 scope.go:122] "RemoveContainer" containerID="97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d"
Nov 03 11:29:00.632580 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.632551    5393 generic.go:356] "Generic (PLEG): container finished" podID="bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3" containerID="71b583eb268847a66e0e1ef9e55f51072a7ac96cc188657c600dd2652dc5bfaa" exitCode=0
Nov 03 11:29:00.632705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.632675    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz" event={"ID":"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3","Type":"ContainerDied","Data":"71b583eb268847a66e0e1ef9e55f51072a7ac96cc188657c600dd2652dc5bfaa"}
Nov 03 11:29:00.634338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.634314    5393 generic.go:356] "Generic (PLEG): container finished" podID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerID="f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397" exitCode=0
Nov 03 11:29:00.634461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.634363    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"
Nov 03 11:29:00.634461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.634373    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" event={"ID":"86a119d8-1185-4b77-b7d4-9f445928fec8","Type":"ContainerDied","Data":"f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397"}
Nov 03 11:29:00.634461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.634414    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" event={"ID":"86a119d8-1185-4b77-b7d4-9f445928fec8","Type":"ContainerDied","Data":"7be5c1d3f6f352ee0c74872d56bd00f1ecf47e7c2c21ac826260e42cd2f8b47c"}
Nov 03 11:29:00.643997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.643975    5393 scope.go:122] "RemoveContainer" containerID="97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d"
Nov 03 11:29:00.644434 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:00.644404    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d\": container with ID starting with 97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d not found: ID does not exist" containerID="97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d"
Nov 03 11:29:00.644546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.644461    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d"} err="failed to get container status \"97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d\": rpc error: code = NotFound desc = could not find container \"97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d\": container with ID starting with 97cae3aad8839ef1dc7df2a001bf8e43a6fd098a95ef07384db5b6d752b4d94d not found: ID does not exist"
Nov 03 11:29:00.644546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.644489    5393 scope.go:122] "RemoveContainer" containerID="f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397"
Nov 03 11:29:00.655389 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.655371    5393 scope.go:122] "RemoveContainer" containerID="f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397"
Nov 03 11:29:00.655703 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:00.655682    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397\": container with ID starting with f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397 not found: ID does not exist" containerID="f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397"
Nov 03 11:29:00.655756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.655710    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397"} err="failed to get container status \"f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397\": rpc error: code = NotFound desc = could not find container \"f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397\": container with ID starting with f800ac4ba92621b0266348849ee566f00e24a26e7704a7ec450a9ddf33a1f397 not found: ID does not exist"
Nov 03 11:29:00.668278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.668249    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"]
Nov 03 11:29:00.672476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.672449    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/simulator-w2-e78c33-kserve-55679b5b88-ds4bb"]
Nov 03 11:29:00.730679 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.723581    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"]
Nov 03 11:29:00.774373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:00.774343    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc"]
Nov 03 11:29:01.302942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:01.302876    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e78c33/distinct-w2-e78c33-kserve-7c4dc75d7b-fsswc" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 11:29:01.811711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:01.811683    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:29:01.950817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:01.950733    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-kube-api-access-r8dft\" (UniqueName: \"kubernetes.io/projected/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-kube-api-access-r8dft\") pod \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") "
Nov 03 11:29:01.951201 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:01.951178    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-openshift-ca\") pod \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") "
Nov 03 11:29:01.951597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:01.951572    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-xks-ca\") pod \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\" (UID: \"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3\") "
Nov 03 11:29:01.959589 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:01.959517    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-openshift-ca" pod "bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3" (UID: "bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:01.968536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:01.968489    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-xks-ca" pod "bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3" (UID: "bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:01.968882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:01.968846    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-kube-api-access-r8dft" pod "bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3" (UID: "bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3"). InnerVolumeSpecName "kube-api-access-r8dft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:02.060108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:02.060063    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8dft\" (UniqueName: \"kubernetes.io/projected/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-kube-api-access-r8dft\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:02.060108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:02.060103    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:02.060108 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:02.060116    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:02.231863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:02.231777    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64742526-7053-46b4-afd4-5043ebbda083" path="/var/lib/kubelet/pods/64742526-7053-46b4-afd4-5043ebbda083/volumes"
Nov 03 11:29:02.232278 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:02.232263    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" path="/var/lib/kubelet/pods/86a119d8-1185-4b77-b7d4-9f445928fec8/volumes"
Nov 03 11:29:02.651814 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:02.651781    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"
Nov 03 11:29:02.652040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:02.651819    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz" event={"ID":"bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3","Type":"ContainerDied","Data":"94a14ac45d00254059a7cab767d3b3f6134261fd2ce47f5a01cd9363fa2b962a"}
Nov 03 11:29:02.652040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:02.651867    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94a14ac45d00254059a7cab767d3b3f6134261fd2ce47f5a01cd9363fa2b962a"
Nov 03 11:29:05.446279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:05.444989    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz" podUID="aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" containerName="payload-processing" containerID="cri-o://16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923" gracePeriod=25
Nov 03 11:29:05.460954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:05.460881    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm" podUID="2ffd7c61-7685-449c-84d1-045d19c6148d" containerName="payload-pre-processing" containerID="cri-o://de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0" gracePeriod=25
Nov 03 11:29:06.993723 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.993689    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-b9ee02"]
Nov 03 11:29:06.994347 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994327    5393 cpu_manager.go:414] "containerMap: removing container" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main"
Nov 03 11:29:06.994402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994350    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main"
Nov 03 11:29:06.994402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994370    5393 cpu_manager.go:414] "containerMap: removing container" podUID="bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3" containerName="revoke-keys"
Nov 03 11:29:06.994402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994378    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3" containerName="revoke-keys"
Nov 03 11:29:06.994504 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994402    5393 cpu_manager.go:414] "containerMap: removing container" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main"
Nov 03 11:29:06.994504 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994410    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main"
Nov 03 11:29:06.994604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994592    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a119d8-1185-4b77-b7d4-9f445928fec8" containerName="main"
Nov 03 11:29:06.994641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994618    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fa9ef1-cff1-4c95-bcea-b79ef3c5c1f9" containerName="istio-proxy"
Nov 03 11:29:06.994641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994633    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="64742526-7053-46b4-afd4-5043ebbda083" containerName="main"
Nov 03 11:29:06.994702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:06.994644    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3" containerName="revoke-keys"
Nov 03 11:29:07.000262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.000238    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-b9ee02"
Nov 03 11:29:07.001285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.001256    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-b9ee02"]
Nov 03 11:29:07.006968 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.004230    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5zm6k\""
Nov 03 11:29:07.163637 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.163603    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8846\" (UniqueName: \"kubernetes.io/projected/a8bf20e1-fb71-4434-ab14-39a06bc8aeff-kube-api-access-k8846\") pod \"test-curl-7600-b9ee02\" (UID: \"a8bf20e1-fb71-4434-ab14-39a06bc8aeff\") " pod="opendatahub/test-curl-7600-b9ee02"
Nov 03 11:29:07.168053 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.168021    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm" podUID="2ffd7c61-7685-449c-84d1-045d19c6148d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:29:07.168523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.168500    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz" podUID="aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:29:07.264994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.264865    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8846\" (UniqueName: \"kubernetes.io/projected/a8bf20e1-fb71-4434-ab14-39a06bc8aeff-kube-api-access-k8846\") pod \"test-curl-7600-b9ee02\" (UID: \"a8bf20e1-fb71-4434-ab14-39a06bc8aeff\") " pod="opendatahub/test-curl-7600-b9ee02"
Nov 03 11:29:07.273251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.273210    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8846\" (UniqueName: \"kubernetes.io/projected/a8bf20e1-fb71-4434-ab14-39a06bc8aeff-kube-api-access-k8846\") pod \"test-curl-7600-b9ee02\" (UID: \"a8bf20e1-fb71-4434-ab14-39a06bc8aeff\") " pod="opendatahub/test-curl-7600-b9ee02"
Nov 03 11:29:07.318213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.318174    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-b9ee02"
Nov 03 11:29:07.479154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.479121    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"]
Nov 03 11:29:07.486132 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.485485    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.493830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.493190    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"]
Nov 03 11:29:07.493830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.493502    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"default-dockercfg-mclmp\""
Nov 03 11:29:07.493830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.493677    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"sub-test-model-836589-kserve-self-signed-certs\""
Nov 03 11:29:07.495643 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.495623    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"kube-root-ca.crt\""
Nov 03 11:29:07.495803 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.495740    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-836589\"/\"openshift-service-ca.crt\""
Nov 03 11:29:07.571310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.571214    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-model-cache\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.571310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.571295    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-tmp-dir\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.571537 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.571422    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee438333-6dc6-40ad-acf8-623a3f1f9164-tls-certs\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.571573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.571531    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-dshm\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.571618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.571601    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-home\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.571739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.571721    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4bn\" (UniqueName: \"kubernetes.io/projected/ee438333-6dc6-40ad-acf8-623a3f1f9164-kube-api-access-tx4bn\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.603279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.603246    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-b9ee02"]
Nov 03 11:29:07.608451 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:29:07.608422    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8bf20e1_fb71_4434_ab14_39a06bc8aeff.slice/crio-a109bc80f348f600d6ecaa399a3444bdabc8d921be84830213690173db86f56c WatchSource:0}: Error finding container a109bc80f348f600d6ecaa399a3444bdabc8d921be84830213690173db86f56c: Status 404 returned error can't find the container with id a109bc80f348f600d6ecaa399a3444bdabc8d921be84830213690173db86f56c
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.673517    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4bn\" (UniqueName: \"kubernetes.io/projected/ee438333-6dc6-40ad-acf8-623a3f1f9164-kube-api-access-tx4bn\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.673666    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-model-cache\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.673772    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-tmp-dir\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.673811    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee438333-6dc6-40ad-acf8-623a3f1f9164-tls-certs\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.673888    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-dshm\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.673974    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-home\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.674485    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-home\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.675331    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-model-cache\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.676222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.675567    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-tmp-dir\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.773020 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.772976    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-dshm\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.773241 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.773041    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee438333-6dc6-40ad-acf8-623a3f1f9164-tls-certs\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.774602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.774567    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4bn\" (UniqueName: \"kubernetes.io/projected/ee438333-6dc6-40ad-acf8-623a3f1f9164-kube-api-access-tx4bn\") pod \"sub-test-model-836589-kserve-5d6565cfc5-bns56\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") " pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:07.781709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.780694    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-b9ee02" event={"ID":"a8bf20e1-fb71-4434-ab14-39a06bc8aeff","Type":"ContainerStarted","Data":"a109bc80f348f600d6ecaa399a3444bdabc8d921be84830213690173db86f56c"}
Nov 03 11:29:07.813892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:07.813855    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:08.097389 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:08.097356    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"]
Nov 03 11:29:08.098408 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:29:08.098376    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee438333_6dc6_40ad_acf8_623a3f1f9164.slice/crio-b560ac0926847d0f867a2cdfd5e66f1d09adc654e6bcb0577df6f02cbdd16e13 WatchSource:0}: Error finding container b560ac0926847d0f867a2cdfd5e66f1d09adc654e6bcb0577df6f02cbdd16e13: Status 404 returned error can't find the container with id b560ac0926847d0f867a2cdfd5e66f1d09adc654e6bcb0577df6f02cbdd16e13
Nov 03 11:29:08.790150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:08.790109    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56" event={"ID":"ee438333-6dc6-40ad-acf8-623a3f1f9164","Type":"ContainerStarted","Data":"49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8"}
Nov 03 11:29:08.790348 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:08.790164    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56" event={"ID":"ee438333-6dc6-40ad-acf8-623a3f1f9164","Type":"ContainerStarted","Data":"b560ac0926847d0f867a2cdfd5e66f1d09adc654e6bcb0577df6f02cbdd16e13"}
Nov 03 11:29:08.792077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:08.792047    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-b9ee02" event={"ID":"a8bf20e1-fb71-4434-ab14-39a06bc8aeff","Type":"ContainerStarted","Data":"be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e"}
Nov 03 11:29:08.806790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:08.806713    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56" podStartSLOduration=1.806696274 podStartE2EDuration="1.806696274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:08.805637772 +0000 UTC m=+3592.710492214" watchObservedRunningTime="2026-11-03 11:29:08.806696274 +0000 UTC m=+3592.711550790"
Nov 03 11:29:08.814262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:08.814236    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:08.822732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:08.822687    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-b9ee02" podStartSLOduration=2.8226744569999997 podStartE2EDuration="2.822674457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:08.820114286 +0000 UTC m=+3592.724968728" watchObservedRunningTime="2026-11-03 11:29:08.822674457 +0000 UTC m=+3592.727528898"
Nov 03 11:29:09.482852 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.482808    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-b9ee02"]
Nov 03 11:29:09.487381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.487355    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-b9ee02"]
Nov 03 11:29:09.639366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.639330    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-ba4287"]
Nov 03 11:29:09.639980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.639963    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a8bf20e1-fb71-4434-ab14-39a06bc8aeff" containerName="test-curl-7600-b9ee02"
Nov 03 11:29:09.640036 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.639980    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bf20e1-fb71-4434-ab14-39a06bc8aeff" containerName="test-curl-7600-b9ee02"
Nov 03 11:29:09.640107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.640097    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bf20e1-fb71-4434-ab14-39a06bc8aeff" containerName="test-curl-7600-b9ee02"
Nov 03 11:29:09.643894 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.643721    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:09.647478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.647448    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-ba4287"]
Nov 03 11:29:09.704503 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.704469    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhz5q\" (UniqueName: \"kubernetes.io/projected/19f53f78-73b9-4b54-a09e-51cf1cafa172-kube-api-access-lhz5q\") pod \"test-curl-7600-ba4287\" (UID: \"19f53f78-73b9-4b54-a09e-51cf1cafa172\") " pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:09.805622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.805530    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhz5q\" (UniqueName: \"kubernetes.io/projected/19f53f78-73b9-4b54-a09e-51cf1cafa172-kube-api-access-lhz5q\") pod \"test-curl-7600-ba4287\" (UID: \"19f53f78-73b9-4b54-a09e-51cf1cafa172\") " pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:09.814311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.814283    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhz5q\" (UniqueName: \"kubernetes.io/projected/19f53f78-73b9-4b54-a09e-51cf1cafa172-kube-api-access-lhz5q\") pod \"test-curl-7600-ba4287\" (UID: \"19f53f78-73b9-4b54-a09e-51cf1cafa172\") " pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:09.962571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:09.962532    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:10.258894 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:10.258866    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-ba4287"]
Nov 03 11:29:10.259600 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:29:10.259569    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19f53f78_73b9_4b54_a09e_51cf1cafa172.slice/crio-04c153204ed9dd357bae4e204b9cb5d5596ea75591130c01003d4492df671c72 WatchSource:0}: Error finding container 04c153204ed9dd357bae4e204b9cb5d5596ea75591130c01003d4492df671c72: Status 404 returned error can't find the container with id 04c153204ed9dd357bae4e204b9cb5d5596ea75591130c01003d4492df671c72
Nov 03 11:29:10.807157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:10.807059    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-ba4287" event={"ID":"19f53f78-73b9-4b54-a09e-51cf1cafa172","Type":"ContainerStarted","Data":"5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00"}
Nov 03 11:29:10.807157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:10.807107    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-ba4287" event={"ID":"19f53f78-73b9-4b54-a09e-51cf1cafa172","Type":"ContainerStarted","Data":"04c153204ed9dd357bae4e204b9cb5d5596ea75591130c01003d4492df671c72"}
Nov 03 11:29:10.807682 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:10.807352    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-b9ee02" podUID="a8bf20e1-fb71-4434-ab14-39a06bc8aeff" containerName="test-curl-7600-b9ee02" containerID="cri-o://be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e" gracePeriod=2
Nov 03 11:29:10.824403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:10.824331    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-ba4287" podStartSLOduration=1.824310544 podStartE2EDuration="1.824310544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:10.82137784 +0000 UTC m=+3594.726232282" watchObservedRunningTime="2026-11-03 11:29:10.824310544 +0000 UTC m=+3594.729164986"
Nov 03 11:29:11.506794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:11.506761    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-ba4287"]
Nov 03 11:29:11.511695 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:11.511664    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-ba4287"]
Nov 03 11:29:12.820111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:12.820070    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-ba4287" podUID="19f53f78-73b9-4b54-a09e-51cf1cafa172" containerName="test-curl-7600-ba4287" containerID="cri-o://5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00" gracePeriod=2
Nov 03 11:29:13.006420 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.006388    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-b9ee02"
Nov 03 11:29:13.089996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.086088    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bf20e1-fb71-4434-ab14-39a06bc8aeff-kube-api-access-k8846\" (UniqueName: \"kubernetes.io/projected/a8bf20e1-fb71-4434-ab14-39a06bc8aeff-kube-api-access-k8846\") pod \"a8bf20e1-fb71-4434-ab14-39a06bc8aeff\" (UID: \"a8bf20e1-fb71-4434-ab14-39a06bc8aeff\") "
Nov 03 11:29:13.092715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.092684    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bf20e1-fb71-4434-ab14-39a06bc8aeff-kube-api-access-k8846" pod "a8bf20e1-fb71-4434-ab14-39a06bc8aeff" (UID: "a8bf20e1-fb71-4434-ab14-39a06bc8aeff"). InnerVolumeSpecName "kube-api-access-k8846". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:13.188188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.188135    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8846\" (UniqueName: \"kubernetes.io/projected/a8bf20e1-fb71-4434-ab14-39a06bc8aeff-kube-api-access-k8846\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:13.192562 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:13.192522    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:29:13.829363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.829327    5393 generic.go:356] "Generic (PLEG): container finished" podID="a8bf20e1-fb71-4434-ab14-39a06bc8aeff" containerID="be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e" exitCode=137
Nov 03 11:29:13.829828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.829374    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-b9ee02"
Nov 03 11:29:13.829828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.829413    5393 scope.go:122] "RemoveContainer" containerID="be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e"
Nov 03 11:29:13.839638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.839618    5393 scope.go:122] "RemoveContainer" containerID="be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e"
Nov 03 11:29:13.840003 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:13.839971    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e\": container with ID starting with be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e not found: ID does not exist" containerID="be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e"
Nov 03 11:29:13.840106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:13.840009    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e"} err="failed to get container status \"be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e\": rpc error: code = NotFound desc = could not find container \"be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e\": container with ID starting with be9aa549111700130ebd2b813dd7e69c917e30fb5b89368dd1173bf2c430385e not found: ID does not exist"
Nov 03 11:29:13.840637 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:13.840614    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-b9ee02\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="a8bf20e1-fb71-4434-ab14-39a06bc8aeff" pod="opendatahub/test-curl-7600-b9ee02"
Nov 03 11:29:14.233245 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:14.233207    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bf20e1-fb71-4434-ab14-39a06bc8aeff" path="/var/lib/kubelet/pods/a8bf20e1-fb71-4434-ab14-39a06bc8aeff/volumes"
Nov 03 11:29:15.055016 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.054986    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:15.117061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.117021    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f53f78-73b9-4b54-a09e-51cf1cafa172-kube-api-access-lhz5q\" (UniqueName: \"kubernetes.io/projected/19f53f78-73b9-4b54-a09e-51cf1cafa172-kube-api-access-lhz5q\") pod \"19f53f78-73b9-4b54-a09e-51cf1cafa172\" (UID: \"19f53f78-73b9-4b54-a09e-51cf1cafa172\") "
Nov 03 11:29:15.119316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.119288    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f53f78-73b9-4b54-a09e-51cf1cafa172-kube-api-access-lhz5q" pod "19f53f78-73b9-4b54-a09e-51cf1cafa172" (UID: "19f53f78-73b9-4b54-a09e-51cf1cafa172"). InnerVolumeSpecName "kube-api-access-lhz5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:15.218822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.218782    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhz5q\" (UniqueName: \"kubernetes.io/projected/19f53f78-73b9-4b54-a09e-51cf1cafa172-kube-api-access-lhz5q\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:15.886704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.886663    5393 generic.go:356] "Generic (PLEG): container finished" podID="19f53f78-73b9-4b54-a09e-51cf1cafa172" containerID="5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00" exitCode=137
Nov 03 11:29:15.887032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.886908    5393 scope.go:122] "RemoveContainer" containerID="5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00"
Nov 03 11:29:15.887301 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.887275    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:15.899237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.899207    5393 scope.go:122] "RemoveContainer" containerID="5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00"
Nov 03 11:29:15.899661 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:15.899638    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00\": container with ID starting with 5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00 not found: ID does not exist" containerID="5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00"
Nov 03 11:29:15.899760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:15.899670    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00"} err="failed to get container status \"5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00\": rpc error: code = NotFound desc = could not find container \"5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00\": container with ID starting with 5c409e4dd78881142ab87758e1debd1aabbaf930c2919a4a824a390cf9ea0b00 not found: ID does not exist"
Nov 03 11:29:15.900725 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:15.900696    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-ba4287\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="19f53f78-73b9-4b54-a09e-51cf1cafa172" pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:16.229152 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:16.229030    5393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-ba4287\" is forbidden: User \"system:node:ip-10-0-134-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-12.us-east-2.compute.internal' and this object" podUID="19f53f78-73b9-4b54-a09e-51cf1cafa172" pod="opendatahub/test-curl-7600-ba4287"
Nov 03 11:29:16.233349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:16.233320    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f53f78-73b9-4b54-a09e-51cf1cafa172" path="/var/lib/kubelet/pods/19f53f78-73b9-4b54-a09e-51cf1cafa172/volumes"
Nov 03 11:29:17.167613 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:17.167575    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm" podUID="2ffd7c61-7685-449c-84d1-045d19c6148d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:29:17.168349 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:17.168323    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz" podUID="aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:29:17.814231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:17.814196    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:17.827689 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:17.827661    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:17.912846 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:17.912813    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:29:18.185224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:18.185187    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"]
Nov 03 11:29:18.185455 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:18.185427    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy" containerID="cri-o://ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d" gracePeriod=30
Nov 03 11:29:18.576510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:18.576411    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:29:18.587315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:18.587288    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:29:21.545272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:21.545237    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:29:21.545681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:21.545293    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:29:22.215524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.215484    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"]
Nov 03 11:29:22.216333 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.216312    5393 cpu_manager.go:414] "containerMap: removing container" podUID="19f53f78-73b9-4b54-a09e-51cf1cafa172" containerName="test-curl-7600-ba4287"
Nov 03 11:29:22.216333 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.216331    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f53f78-73b9-4b54-a09e-51cf1cafa172" containerName="test-curl-7600-ba4287"
Nov 03 11:29:22.216479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.216426    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f53f78-73b9-4b54-a09e-51cf1cafa172" containerName="test-curl-7600-ba4287"
Nov 03 11:29:22.221414 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.221392    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.226849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.226656    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:29:22.226849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.226709    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:29:22.238489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.237884    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"]
Nov 03 11:29:22.250237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.250192    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.250381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.250249    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6dsv\" (UniqueName: \"kubernetes.io/projected/08db93e1-7c7f-4bf7-9042-4bccf397b914-kube-api-access-q6dsv\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.250434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.250381    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.351974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.351935    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.351974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.351987    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6dsv\" (UniqueName: \"kubernetes.io/projected/08db93e1-7c7f-4bf7-9042-4bccf397b914-kube-api-access-q6dsv\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.352246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.352026    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.352727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.352701    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.354589 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.354565    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-xks-ca\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.360600 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.360575    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6dsv\" (UniqueName: \"kubernetes.io/projected/08db93e1-7c7f-4bf7-9042-4bccf397b914-kube-api-access-q6dsv\") pod \"maas-api-revoke-sub-604aa2a3-x6n7p\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.540909 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.540593    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:22.832500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.832467    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"]
Nov 03 11:29:22.941352 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:22.941316    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p" event={"ID":"08db93e1-7c7f-4bf7-9042-4bccf397b914","Type":"ContainerStarted","Data":"e57f4adbd74403626ed984cf842f380d077576a3c6feea2783186f498290922f"}
Nov 03 11:29:23.531328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.531302    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 11:29:23.580032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.579999    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-socket\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.580249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580112    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e670692d-bbc2-4941-b905-a68856f100b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e670692d-bbc2-4941-b905-a68856f100b4-istio-podinfo\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.580249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580236    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-envoy\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.580385 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580316    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-socket" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:23.580450 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580432    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e670692d-bbc2-4941-b905-a68856f100b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e670692d-bbc2-4941-b905-a68856f100b4-istiod-ca-cert\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.580569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580552    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-certs\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.580635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580615    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-kube-api-access-j9mcp\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-kube-api-access-j9mcp\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.580697 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580640    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-istio-token\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-istio-token\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.580697 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580675    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-credential-socket\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.580809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.580718    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-data\") pod \"e670692d-bbc2-4941-b905-a68856f100b4\" (UID: \"e670692d-bbc2-4941-b905-a68856f100b4\") "
Nov 03 11:29:23.581030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.581001    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-certs" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:23.581397 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.581373    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.581633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.581402    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.581633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.581456    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e670692d-bbc2-4941-b905-a68856f100b4-istiod-ca-cert" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:23.582194 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.582167    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-data" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:23.582317 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.582270    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-credential-socket" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:23.583426 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.583397    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e670692d-bbc2-4941-b905-a68856f100b4-istio-podinfo" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:29:23.583665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.583635    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-envoy" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:29:23.584671 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.584640    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-kube-api-access-j9mcp" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "kube-api-access-j9mcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:23.591588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.591560    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-istio-token" pod "e670692d-bbc2-4941-b905-a68856f100b4" (UID: "e670692d-bbc2-4941-b905-a68856f100b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:23.682622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.682586    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e670692d-bbc2-4941-b905-a68856f100b4-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.682622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.682620    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.682622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.682630    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e670692d-bbc2-4941-b905-a68856f100b4-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.682892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.682639    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9mcp\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-kube-api-access-j9mcp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.682892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.682649    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e670692d-bbc2-4941-b905-a68856f100b4-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.682892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.682657    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.682892 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.682667    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e670692d-bbc2-4941-b905-a68856f100b4-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:23.949932 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.949895    5393 generic.go:356] "Generic (PLEG): container finished" podID="08db93e1-7c7f-4bf7-9042-4bccf397b914" containerID="a21bcc7313c0dd40029843f67717fb3931b406ee1ff0083da4ed473a002abe8d" exitCode=0
Nov 03 11:29:23.950395 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.949987    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p" event={"ID":"08db93e1-7c7f-4bf7-9042-4bccf397b914","Type":"ContainerDied","Data":"a21bcc7313c0dd40029843f67717fb3931b406ee1ff0083da4ed473a002abe8d"}
Nov 03 11:29:23.951374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.951351    5393 generic.go:356] "Generic (PLEG): container finished" podID="e670692d-bbc2-4941-b905-a68856f100b4" containerID="ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d" exitCode=0
Nov 03 11:29:23.951496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.951391    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"
Nov 03 11:29:23.951496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.951432    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" event={"ID":"e670692d-bbc2-4941-b905-a68856f100b4","Type":"ContainerDied","Data":"ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d"}
Nov 03 11:29:23.951496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.951488    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw" event={"ID":"e670692d-bbc2-4941-b905-a68856f100b4","Type":"ContainerDied","Data":"34324785ecaf2a42d30232b1ea380490357cb695f8a28389817a00fafe6fd22f"}
Nov 03 11:29:23.951607 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.951511    5393 scope.go:122] "RemoveContainer" containerID="ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d"
Nov 03 11:29:23.962510 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.962487    5393 scope.go:122] "RemoveContainer" containerID="ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d"
Nov 03 11:29:23.962769 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:23.962751    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d\": container with ID starting with ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d not found: ID does not exist" containerID="ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d"
Nov 03 11:29:23.962833 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:23.962782    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d"} err="failed to get container status \"ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d\": rpc error: code = NotFound desc = could not find container \"ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d\": container with ID starting with ed5fb010a2b60f95dfe7545de0eb972626d9a7e6db34c7c533536da34212697d not found: ID does not exist"
Nov 03 11:29:24.001447 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:24.001412    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"]
Nov 03 11:29:24.005280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:24.005249    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e92a08-openshift-default-7694cb68f-xz4cw"]
Nov 03 11:29:24.231522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:24.231425    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e670692d-bbc2-4941-b905-a68856f100b4" path="/var/lib/kubelet/pods/e670692d-bbc2-4941-b905-a68856f100b4/volumes"
Nov 03 11:29:25.203675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.203645    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:25.310724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.310630    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-xks-ca\") pod \"08db93e1-7c7f-4bf7-9042-4bccf397b914\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") "
Nov 03 11:29:25.310724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.310686    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-openshift-ca\") pod \"08db93e1-7c7f-4bf7-9042-4bccf397b914\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") "
Nov 03 11:29:25.310997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.310731    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08db93e1-7c7f-4bf7-9042-4bccf397b914-kube-api-access-q6dsv\" (UniqueName: \"kubernetes.io/projected/08db93e1-7c7f-4bf7-9042-4bccf397b914-kube-api-access-q6dsv\") pod \"08db93e1-7c7f-4bf7-9042-4bccf397b914\" (UID: \"08db93e1-7c7f-4bf7-9042-4bccf397b914\") "
Nov 03 11:29:25.311120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.311094    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-openshift-ca" pod "08db93e1-7c7f-4bf7-9042-4bccf397b914" (UID: "08db93e1-7c7f-4bf7-9042-4bccf397b914"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:25.311542 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.311513    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:25.313167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.313143    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-xks-ca" pod "08db93e1-7c7f-4bf7-9042-4bccf397b914" (UID: "08db93e1-7c7f-4bf7-9042-4bccf397b914"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:25.313280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.313190    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08db93e1-7c7f-4bf7-9042-4bccf397b914-kube-api-access-q6dsv" pod "08db93e1-7c7f-4bf7-9042-4bccf397b914" (UID: "08db93e1-7c7f-4bf7-9042-4bccf397b914"). InnerVolumeSpecName "kube-api-access-q6dsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:25.413013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.412981    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08db93e1-7c7f-4bf7-9042-4bccf397b914-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:25.413013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.413009    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6dsv\" (UniqueName: \"kubernetes.io/projected/08db93e1-7c7f-4bf7-9042-4bccf397b914-kube-api-access-q6dsv\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:25.741103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.741076    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 11:29:25.771865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.771837    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 11:29:25.817069 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.816979    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-plugins-config-volume\") pod \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\" (UID: \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\") "
Nov 03 11:29:25.817249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.817084    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffd7c61-7685-449c-84d1-045d19c6148d-kube-api-access-2f9w2\" (UniqueName: \"kubernetes.io/projected/2ffd7c61-7685-449c-84d1-045d19c6148d-kube-api-access-2f9w2\") pod \"2ffd7c61-7685-449c-84d1-045d19c6148d\" (UID: \"2ffd7c61-7685-449c-84d1-045d19c6148d\") "
Nov 03 11:29:25.817335 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.817284    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ffd7c61-7685-449c-84d1-045d19c6148d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffd7c61-7685-449c-84d1-045d19c6148d-plugins-config-volume\") pod \"2ffd7c61-7685-449c-84d1-045d19c6148d\" (UID: \"2ffd7c61-7685-449c-84d1-045d19c6148d\") "
Nov 03 11:29:25.817415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.817396    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-plugins-config-volume" pod "aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" (UID: "aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:25.817473 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.817454    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-kube-api-access-984c6\" (UniqueName: \"kubernetes.io/projected/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-kube-api-access-984c6\") pod \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\" (UID: \"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899\") "
Nov 03 11:29:25.817832 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.817807    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffd7c61-7685-449c-84d1-045d19c6148d-plugins-config-volume" pod "2ffd7c61-7685-449c-84d1-045d19c6148d" (UID: "2ffd7c61-7685-449c-84d1-045d19c6148d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:25.818040 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.818021    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ffd7c61-7685-449c-84d1-045d19c6148d-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:25.818287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.818045    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:25.819881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.819857    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-kube-api-access-984c6" pod "aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" (UID: "aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899"). InnerVolumeSpecName "kube-api-access-984c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:25.820188 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.820162    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffd7c61-7685-449c-84d1-045d19c6148d-kube-api-access-2f9w2" pod "2ffd7c61-7685-449c-84d1-045d19c6148d" (UID: "2ffd7c61-7685-449c-84d1-045d19c6148d"). InnerVolumeSpecName "kube-api-access-2f9w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:25.919843 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.919805    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-984c6\" (UniqueName: \"kubernetes.io/projected/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899-kube-api-access-984c6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:25.919843 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.919844    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9w2\" (UniqueName: \"kubernetes.io/projected/2ffd7c61-7685-449c-84d1-045d19c6148d-kube-api-access-2f9w2\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:25.978804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.978765    5393 generic.go:356] "Generic (PLEG): container finished" podID="aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" containerID="16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923" exitCode=0
Nov 03 11:29:25.979012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.978878    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"
Nov 03 11:29:25.979012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.978909    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz" event={"ID":"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899","Type":"ContainerDied","Data":"16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923"}
Nov 03 11:29:25.979012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.979011    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz" event={"ID":"aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899","Type":"ContainerDied","Data":"5dc97fa03bd2373c05e000729650f1aa9fde579fa935aef604fe3fbbc9fb0373"}
Nov 03 11:29:25.979195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.979041    5393 scope.go:122] "RemoveContainer" containerID="16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923"
Nov 03 11:29:25.980490 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.980471    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"
Nov 03 11:29:25.980602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.980490    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p" event={"ID":"08db93e1-7c7f-4bf7-9042-4bccf397b914","Type":"ContainerDied","Data":"e57f4adbd74403626ed984cf842f380d077576a3c6feea2783186f498290922f"}
Nov 03 11:29:25.980602 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.980534    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e57f4adbd74403626ed984cf842f380d077576a3c6feea2783186f498290922f"
Nov 03 11:29:25.982205 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.982177    5393 generic.go:356] "Generic (PLEG): container finished" podID="2ffd7c61-7685-449c-84d1-045d19c6148d" containerID="de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0" exitCode=0
Nov 03 11:29:25.982351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.982224    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"
Nov 03 11:29:25.982351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.982231    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm" event={"ID":"2ffd7c61-7685-449c-84d1-045d19c6148d","Type":"ContainerDied","Data":"de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0"}
Nov 03 11:29:25.982351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.982267    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm" event={"ID":"2ffd7c61-7685-449c-84d1-045d19c6148d","Type":"ContainerDied","Data":"d7d6155810983e91726d6d3fafa150bdc8ad19a7c67fcf03ccb92b59e98c0134"}
Nov 03 11:29:25.991419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.991393    5393 scope.go:122] "RemoveContainer" containerID="16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923"
Nov 03 11:29:25.991790 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:25.991763    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923\": container with ID starting with 16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923 not found: ID does not exist" containerID="16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923"
Nov 03 11:29:25.991853 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.991803    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923"} err="failed to get container status \"16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923\": rpc error: code = NotFound desc = could not find container \"16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923\": container with ID starting with 16a001ebc4048359c96846fbb7060e355c71841e81e13c470f5ff92880fcd923 not found: ID does not exist"
Nov 03 11:29:25.991853 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:25.991833    5393 scope.go:122] "RemoveContainer" containerID="de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0"
Nov 03 11:29:26.002328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.002306    5393 scope.go:122] "RemoveContainer" containerID="de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0"
Nov 03 11:29:26.002688 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:26.002664    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0\": container with ID starting with de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0 not found: ID does not exist" containerID="de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0"
Nov 03 11:29:26.002781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.002697    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0"} err="failed to get container status \"de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0\": rpc error: code = NotFound desc = could not find container \"de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0\": container with ID starting with de93244108952140f96e7324a449b3830904dea41f45e5293c68c5dcc85555e0 not found: ID does not exist"
Nov 03 11:29:26.009693 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.009669    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"]
Nov 03 11:29:26.018669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.018642    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e92a08-cb5756c95-mb7pz"]
Nov 03 11:29:26.034162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.034132    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"]
Nov 03 11:29:26.037621 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.037594    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e92a08-5d64d7fdf5-9prtm"]
Nov 03 11:29:26.232512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.232476    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ffd7c61-7685-449c-84d1-045d19c6148d" path="/var/lib/kubelet/pods/2ffd7c61-7685-449c-84d1-045d19c6148d/volumes"
Nov 03 11:29:26.232939 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.232882    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" path="/var/lib/kubelet/pods/aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899/volumes"
Nov 03 11:29:26.317813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.317774    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"]
Nov 03 11:29:26.318479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318457    5393 cpu_manager.go:414] "containerMap: removing container" podUID="08db93e1-7c7f-4bf7-9042-4bccf397b914" containerName="revoke-keys"
Nov 03 11:29:26.318595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318480    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08db93e1-7c7f-4bf7-9042-4bccf397b914" containerName="revoke-keys"
Nov 03 11:29:26.318595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318493    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy"
Nov 03 11:29:26.318595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318500    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy"
Nov 03 11:29:26.318595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318539    5393 cpu_manager.go:414] "containerMap: removing container" podUID="aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" containerName="payload-processing"
Nov 03 11:29:26.318595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318547    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" containerName="payload-processing"
Nov 03 11:29:26.318595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318580    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2ffd7c61-7685-449c-84d1-045d19c6148d" containerName="payload-pre-processing"
Nov 03 11:29:26.318595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318587    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffd7c61-7685-449c-84d1-045d19c6148d" containerName="payload-pre-processing"
Nov 03 11:29:26.318987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318745    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="08db93e1-7c7f-4bf7-9042-4bccf397b914" containerName="revoke-keys"
Nov 03 11:29:26.318987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318761    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e670692d-bbc2-4941-b905-a68856f100b4" containerName="istio-proxy"
Nov 03 11:29:26.318987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318776    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6c76ed-60c6-4ce1-a9c7-8acc2e0be899" containerName="payload-processing"
Nov 03 11:29:26.318987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.318790    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffd7c61-7685-449c-84d1-045d19c6148d" containerName="payload-pre-processing"
Nov 03 11:29:26.323934 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.323894    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.341962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.336431    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"]
Nov 03 11:29:26.374036 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.374008    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-9c53b2-openshift-default-dockercfg-jmr5g\""
Nov 03 11:29:26.425951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.425886    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.426152 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.425955    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-istio-token\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.426152 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.426051    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3761a592-894d-4ce7-af27-594c7498dc45-istiod-ca-cert\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.426263 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.426228    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3761a592-894d-4ce7-af27-594c7498dc45-istio-podinfo\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.426322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.426305    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm42w\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-kube-api-access-zm42w\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.426369 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.426343    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-credential-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.426435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.426366    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-certs\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.426546 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.426514    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-data\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.426699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.426615    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-envoy\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.527810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.527724    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.527810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.527774    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-istio-token\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.527810 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.527796    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3761a592-894d-4ce7-af27-594c7498dc45-istiod-ca-cert\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.527867    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3761a592-894d-4ce7-af27-594c7498dc45-istio-podinfo\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.527894    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm42w\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-kube-api-access-zm42w\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.527957    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-credential-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.527991    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-certs\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.528035    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-data\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.528118    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-envoy\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.528145    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.528428    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-certs\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.528463    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-data\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.528602    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-credential-socket\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.528668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.528613    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3761a592-894d-4ce7-af27-594c7498dc45-istiod-ca-cert\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.530554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.530532    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-envoy\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.530657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.530622    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3761a592-894d-4ce7-af27-594c7498dc45-istio-podinfo\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.535603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.535583    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-istio-token\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.536012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.535991    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm42w\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-kube-api-access-zm42w\") pod \"e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") " pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.651026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.650987    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:26.942607 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.942576    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"]
Nov 03 11:29:26.943279 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:29:26.943253    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3761a592_894d_4ce7_af27_594c7498dc45.slice/crio-5eab6dd9c929930bd92632c2aca3b1560f11aafece2594c828dd3d24212fb8c1 WatchSource:0}: Error finding container 5eab6dd9c929930bd92632c2aca3b1560f11aafece2594c828dd3d24212fb8c1: Status 404 returned error can't find the container with id 5eab6dd9c929930bd92632c2aca3b1560f11aafece2594c828dd3d24212fb8c1
Nov 03 11:29:26.946060 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.946025    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:29:26.946135 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.946098    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:29:26.946171 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.946132    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:29:26.994599 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:26.994564    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" event={"ID":"3761a592-894d-4ce7-af27-594c7498dc45","Type":"ContainerStarted","Data":"5eab6dd9c929930bd92632c2aca3b1560f11aafece2594c828dd3d24212fb8c1"}
Nov 03 11:29:28.018044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:28.017999    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" event={"ID":"3761a592-894d-4ce7-af27-594c7498dc45","Type":"ContainerStarted","Data":"0134ce9ac2cc24c9414b5a35d8713b10f7dc256341d90c596098234a95c07326"}
Nov 03 11:29:28.042091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:28.042042    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" podStartSLOduration=2.042027073 podStartE2EDuration="2.042027073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:28.040702417 +0000 UTC m=+3611.945556870" watchObservedRunningTime="2026-11-03 11:29:28.042027073 +0000 UTC m=+3611.946881516"
Nov 03 11:29:28.651710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:28.651671    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:28.653496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:28.653468    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:29:28.653633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:28.653523    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Nov 03 11:29:28.821217 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:28.821181    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"]
Nov 03 11:29:29.651788 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:29.651747    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:29:29.652198 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:29.651806    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Nov 03 11:29:30.652214 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:30.652174    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:29:30.652708 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:30.652237    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Nov 03 11:29:31.658953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:31.658484    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:29:31.658953 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:31.658548    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Nov 03 11:29:32.651582 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:32.651539    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:29:32.651801 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:32.651615    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Nov 03 11:29:33.652127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:33.652085    5393 patch_prober.go:29] interesting pod/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused" start-of-body=
Nov 03 11:29:33.652625 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:33.652168    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.58:15021/healthz/ready\": dial tcp 10.134.1.58:15021: connect: connection refused"
Nov 03 11:29:33.822299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:33.822234    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t" podUID="69b17534-222e-4854-8bfc-7bd86b8af595" containerName="payload-processing" containerID="cri-o://59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9" gracePeriod=25
Nov 03 11:29:34.656416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:34.656385    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:34.656905 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:34.656882    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:34.657705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:34.657686    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:29:34.834802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:34.834759    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t" podUID="69b17534-222e-4854-8bfc-7bd86b8af595" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:29:41.755376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.755337    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"]
Nov 03 11:29:41.759627 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.759608    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.767080 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.767052    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-ef1662-openshift-default-dockercfg-c2trg\""
Nov 03 11:29:41.770650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.770620    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"]
Nov 03 11:29:41.934200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934165    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.934412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934312    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-data\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.934472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934437    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwck\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-kube-api-access-hfwck\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.934525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934482    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-istio-token\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.934619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934553    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-envoy\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.934619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934609    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d52307-b059-47eb-a13e-c357fb01e355-istio-podinfo\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.934704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934647    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-certs\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.934770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934747    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-credential-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:41.934844 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:41.934826    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d52307-b059-47eb-a13e-c357fb01e355-istiod-ca-cert\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036234    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036322    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-data\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036397    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwck\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-kube-api-access-hfwck\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036433    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-istio-token\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036475    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-envoy\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036675    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d52307-b059-47eb-a13e-c357fb01e355-istio-podinfo\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036748    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036773    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-certs\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036841    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-credential-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036790    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-data\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.036956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.036911    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d52307-b059-47eb-a13e-c357fb01e355-istiod-ca-cert\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.037383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.037013    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-certs\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.037383 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.037094    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-credential-socket\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.037489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.037470    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d52307-b059-47eb-a13e-c357fb01e355-istiod-ca-cert\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.039162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.039141    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-envoy\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.039635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.039611    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d52307-b059-47eb-a13e-c357fb01e355-istio-podinfo\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.045229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.045204    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-istio-token\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.045507 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.045485    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwck\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-kube-api-access-hfwck\") pod \"e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") " pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.075567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.075536    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:42.366372 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.366339    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"]
Nov 03 11:29:42.367529 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:29:42.367504    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d52307_b059_47eb_a13e_c357fb01e355.slice/crio-f51c3cb9963b4744d442ec406bede0bc98ea9e803e7c92dd9a5e566ac7d9cba9 WatchSource:0}: Error finding container f51c3cb9963b4744d442ec406bede0bc98ea9e803e7c92dd9a5e566ac7d9cba9: Status 404 returned error can't find the container with id f51c3cb9963b4744d442ec406bede0bc98ea9e803e7c92dd9a5e566ac7d9cba9
Nov 03 11:29:42.370332 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.370294    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:29:42.370430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.370372    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:29:42.370430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:42.370403    5393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 11:29:43.168329 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:43.168297    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" event={"ID":"f5d52307-b059-47eb-a13e-c357fb01e355","Type":"ContainerStarted","Data":"a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6"}
Nov 03 11:29:43.168830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:43.168348    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" event={"ID":"f5d52307-b059-47eb-a13e-c357fb01e355","Type":"ContainerStarted","Data":"f51c3cb9963b4744d442ec406bede0bc98ea9e803e7c92dd9a5e566ac7d9cba9"}
Nov 03 11:29:43.206945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:43.193523    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" podStartSLOduration=2.19350245 podStartE2EDuration="2.19350245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:43.189459153 +0000 UTC m=+3627.094313598" watchObservedRunningTime="2026-11-03 11:29:43.19350245 +0000 UTC m=+3627.098356884"
Nov 03 11:29:43.217222 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:43.217182    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:29:44.081257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.081210    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:44.088061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.088026    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 03 11:29:44.088344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.088314    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 03 11:29:44.575180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.575139    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"]
Nov 03 11:29:44.579711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.579685    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.608045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.606639    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:29:44.608045 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.606713    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:29:44.614499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.614465    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.614868 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.614846    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.615107 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.615089    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdckb\" (UniqueName: \"kubernetes.io/projected/506a91f8-4a09-4bb0-99e7-d351273500d9-kube-api-access-sdckb\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.622963 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.620111    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"]
Nov 03 11:29:44.716090 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.716055    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.716269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.716138    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdckb\" (UniqueName: \"kubernetes.io/projected/506a91f8-4a09-4bb0-99e7-d351273500d9-kube-api-access-sdckb\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.716326 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.716310    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.716817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.716798    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.719084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.719062    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.725381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.725358    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdckb\" (UniqueName: \"kubernetes.io/projected/506a91f8-4a09-4bb0-99e7-d351273500d9-kube-api-access-sdckb\") pod \"maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:44.834359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.834264    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t" podUID="69b17534-222e-4854-8bfc-7bd86b8af595" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:29:44.930355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:44.930320    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:45.076900 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:45.076873    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 03 11:29:45.077427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:45.077126    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 03 11:29:45.240282 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:45.240255    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"]
Nov 03 11:29:45.241728 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:29:45.241697    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506a91f8_4a09_4bb0_99e7_d351273500d9.slice/crio-8b45463b91f5cc1c6faf1b96b26a96ef978641eaf80ea7c3914f06cdee284f76 WatchSource:0}: Error finding container 8b45463b91f5cc1c6faf1b96b26a96ef978641eaf80ea7c3914f06cdee284f76: Status 404 returned error can't find the container with id 8b45463b91f5cc1c6faf1b96b26a96ef978641eaf80ea7c3914f06cdee284f76
Nov 03 11:29:46.086018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:46.083482    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 03 11:29:46.086018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:46.083544    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 03 11:29:46.215294 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:46.215251    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c" event={"ID":"506a91f8-4a09-4bb0-99e7-d351273500d9","Type":"ContainerStarted","Data":"538eba64598cd85f3f9656dd2c66f8d831c7b10bd513890febb64f150b36545f"}
Nov 03 11:29:46.215484 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:46.215344    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c" event={"ID":"506a91f8-4a09-4bb0-99e7-d351273500d9","Type":"ContainerStarted","Data":"8b45463b91f5cc1c6faf1b96b26a96ef978641eaf80ea7c3914f06cdee284f76"}
Nov 03 11:29:46.232446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:46.232397    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c" podStartSLOduration=2.232383591 podStartE2EDuration="2.232383591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:29:46.230130914 +0000 UTC m=+3630.134985356" watchObservedRunningTime="2026-11-03 11:29:46.232383591 +0000 UTC m=+3630.137238033"
Nov 03 11:29:47.076529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:47.076498    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 03 11:29:47.076728 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:47.076546    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 03 11:29:47.224516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:47.224478    5393 generic.go:356] "Generic (PLEG): container finished" podID="506a91f8-4a09-4bb0-99e7-d351273500d9" containerID="538eba64598cd85f3f9656dd2c66f8d831c7b10bd513890febb64f150b36545f" exitCode=0
Nov 03 11:29:47.225007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:47.224562    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c" event={"ID":"506a91f8-4a09-4bb0-99e7-d351273500d9","Type":"ContainerDied","Data":"538eba64598cd85f3f9656dd2c66f8d831c7b10bd513890febb64f150b36545f"}
Nov 03 11:29:48.076276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.076242    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 03 11:29:48.076513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.076302    5393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 03 11:29:48.473240 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.473212    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:48.570720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.570680    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506a91f8-4a09-4bb0-99e7-d351273500d9-kube-api-access-sdckb\" (UniqueName: \"kubernetes.io/projected/506a91f8-4a09-4bb0-99e7-d351273500d9-kube-api-access-sdckb\") pod \"506a91f8-4a09-4bb0-99e7-d351273500d9\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") "
Nov 03 11:29:48.570952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.570767    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-openshift-ca\") pod \"506a91f8-4a09-4bb0-99e7-d351273500d9\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") "
Nov 03 11:29:48.570952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.570831    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-xks-ca\") pod \"506a91f8-4a09-4bb0-99e7-d351273500d9\" (UID: \"506a91f8-4a09-4bb0-99e7-d351273500d9\") "
Nov 03 11:29:48.571247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.571222    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-openshift-ca" pod "506a91f8-4a09-4bb0-99e7-d351273500d9" (UID: "506a91f8-4a09-4bb0-99e7-d351273500d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:48.571498 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.571480    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:48.573299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.573275    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-xks-ca" pod "506a91f8-4a09-4bb0-99e7-d351273500d9" (UID: "506a91f8-4a09-4bb0-99e7-d351273500d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:29:48.573417 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.573401    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506a91f8-4a09-4bb0-99e7-d351273500d9-kube-api-access-sdckb" pod "506a91f8-4a09-4bb0-99e7-d351273500d9" (UID: "506a91f8-4a09-4bb0-99e7-d351273500d9"). InnerVolumeSpecName "kube-api-access-sdckb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:48.672980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.672939    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdckb\" (UniqueName: \"kubernetes.io/projected/506a91f8-4a09-4bb0-99e7-d351273500d9-kube-api-access-sdckb\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:48.672980 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:48.672983    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506a91f8-4a09-4bb0-99e7-d351273500d9-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:49.140137 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:49.140102    5393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:49.140514 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:49.140495    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:49.141367 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:49.141344    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:29:49.241502 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:49.241474    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"
Nov 03 11:29:49.241502 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:49.241472    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c" event={"ID":"506a91f8-4a09-4bb0-99e7-d351273500d9","Type":"ContainerDied","Data":"8b45463b91f5cc1c6faf1b96b26a96ef978641eaf80ea7c3914f06cdee284f76"}
Nov 03 11:29:49.241758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:49.241544    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b45463b91f5cc1c6faf1b96b26a96ef978641eaf80ea7c3914f06cdee284f76"
Nov 03 11:29:49.472588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:49.472493    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"]
Nov 03 11:29:54.100133 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.100104    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 11:29:54.151165 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.151136    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b17534-222e-4854-8bfc-7bd86b8af595-kube-api-access-jd89c\" (UniqueName: \"kubernetes.io/projected/69b17534-222e-4854-8bfc-7bd86b8af595-kube-api-access-jd89c\") pod \"69b17534-222e-4854-8bfc-7bd86b8af595\" (UID: \"69b17534-222e-4854-8bfc-7bd86b8af595\") "
Nov 03 11:29:54.151357 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.151288    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b17534-222e-4854-8bfc-7bd86b8af595-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b17534-222e-4854-8bfc-7bd86b8af595-plugins-config-volume\") pod \"69b17534-222e-4854-8bfc-7bd86b8af595\" (UID: \"69b17534-222e-4854-8bfc-7bd86b8af595\") "
Nov 03 11:29:54.151753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.151720    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b17534-222e-4854-8bfc-7bd86b8af595-plugins-config-volume" pod "69b17534-222e-4854-8bfc-7bd86b8af595" (UID: "69b17534-222e-4854-8bfc-7bd86b8af595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:29:54.152162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.152084    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b17534-222e-4854-8bfc-7bd86b8af595-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:54.153705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.153678    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b17534-222e-4854-8bfc-7bd86b8af595-kube-api-access-jd89c" pod "69b17534-222e-4854-8bfc-7bd86b8af595" (UID: "69b17534-222e-4854-8bfc-7bd86b8af595"). InnerVolumeSpecName "kube-api-access-jd89c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:29:54.253341 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.253310    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd89c\" (UniqueName: \"kubernetes.io/projected/69b17534-222e-4854-8bfc-7bd86b8af595-kube-api-access-jd89c\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:29:54.278545 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.278507    5393 generic.go:356] "Generic (PLEG): container finished" podID="69b17534-222e-4854-8bfc-7bd86b8af595" containerID="59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9" exitCode=0
Nov 03 11:29:54.278742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.278555    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"
Nov 03 11:29:54.278742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.278598    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t" event={"ID":"69b17534-222e-4854-8bfc-7bd86b8af595","Type":"ContainerDied","Data":"59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9"}
Nov 03 11:29:54.278742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.278671    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t" event={"ID":"69b17534-222e-4854-8bfc-7bd86b8af595","Type":"ContainerDied","Data":"e9fb1812d4ca4c283136de1633ba5e63daa5cfa766ddf1df7ee32b65c2268a96"}
Nov 03 11:29:54.278742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.278701    5393 scope.go:122] "RemoveContainer" containerID="59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9"
Nov 03 11:29:54.289948 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.289904    5393 scope.go:122] "RemoveContainer" containerID="59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9"
Nov 03 11:29:54.290275 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:29:54.290253    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9\": container with ID starting with 59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9 not found: ID does not exist" containerID="59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9"
Nov 03 11:29:54.290353 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.290284    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9"} err="failed to get container status \"59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9\": rpc error: code = NotFound desc = could not find container \"59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9\": container with ID starting with 59467dfc2fa8fb36bfc1da879fbfb4b80e17debba335f1edfce7db3566a292c9 not found: ID does not exist"
Nov 03 11:29:54.296826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.296802    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"]
Nov 03 11:29:54.302070 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:54.302046    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-fcc2ce-694c767579-9bf9t"]
Nov 03 11:29:56.232418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:56.232383    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b17534-222e-4854-8bfc-7bd86b8af595" path="/var/lib/kubelet/pods/69b17534-222e-4854-8bfc-7bd86b8af595/volumes"
Nov 03 11:29:57.672348 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.672314    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"]
Nov 03 11:29:57.672946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.672931    5393 cpu_manager.go:414] "containerMap: removing container" podUID="506a91f8-4a09-4bb0-99e7-d351273500d9" containerName="revoke-keys"
Nov 03 11:29:57.673018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.672947    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506a91f8-4a09-4bb0-99e7-d351273500d9" containerName="revoke-keys"
Nov 03 11:29:57.673018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.672997    5393 cpu_manager.go:414] "containerMap: removing container" podUID="69b17534-222e-4854-8bfc-7bd86b8af595" containerName="payload-processing"
Nov 03 11:29:57.673018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.673002    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b17534-222e-4854-8bfc-7bd86b8af595" containerName="payload-processing"
Nov 03 11:29:57.673143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.673098    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="506a91f8-4a09-4bb0-99e7-d351273500d9" containerName="revoke-keys"
Nov 03 11:29:57.673143 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.673117    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b17534-222e-4854-8bfc-7bd86b8af595" containerName="payload-processing"
Nov 03 11:29:57.678106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.678077    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.686180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.686150    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:29:57.686574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.686553    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:29:57.687854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.687828    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"]
Nov 03 11:29:57.811663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.811629    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.811853 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.811719    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.811853 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.811797    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jlp\" (UniqueName: \"kubernetes.io/projected/b03af0f2-633e-48b4-ac32-f78749a60da3-kube-api-access-22jlp\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.913513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.913478    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.913513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.913519    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.913794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.913573    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jlp\" (UniqueName: \"kubernetes.io/projected/b03af0f2-633e-48b4-ac32-f78749a60da3-kube-api-access-22jlp\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.914329 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.914301    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.916212 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.916183    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:57.922641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:57.922586    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jlp\" (UniqueName: \"kubernetes.io/projected/b03af0f2-633e-48b4-ac32-f78749a60da3-kube-api-access-22jlp\") pod \"maas-api-revoke-sub-6869004a-cs4w7\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:58.000670 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:58.000632    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:29:58.283433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:58.283402    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"]
Nov 03 11:29:58.284840 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:29:58.284799    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03af0f2_633e_48b4_ac32_f78749a60da3.slice/crio-d947062c27e613656c510e632d0f6cd558984a44281527ece09b782c3581425e WatchSource:0}: Error finding container d947062c27e613656c510e632d0f6cd558984a44281527ece09b782c3581425e: Status 404 returned error can't find the container with id d947062c27e613656c510e632d0f6cd558984a44281527ece09b782c3581425e
Nov 03 11:29:58.310996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:58.310965    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7" event={"ID":"b03af0f2-633e-48b4-ac32-f78749a60da3","Type":"ContainerStarted","Data":"d947062c27e613656c510e632d0f6cd558984a44281527ece09b782c3581425e"}
Nov 03 11:29:59.390358 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:59.390323    5393 generic.go:356] "Generic (PLEG): container finished" podID="b03af0f2-633e-48b4-ac32-f78749a60da3" containerID="6a032dee8d7f2308e53f83264fb6903956b25760cf1f22eaa0110dc749f85a48" exitCode=0
Nov 03 11:29:59.390864 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:29:59.390398    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7" event={"ID":"b03af0f2-633e-48b4-ac32-f78749a60da3","Type":"ContainerDied","Data":"6a032dee8d7f2308e53f83264fb6903956b25760cf1f22eaa0110dc749f85a48"}
Nov 03 11:30:00.140157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.140113    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"]
Nov 03 11:30:00.144961 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.144937    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:00.145846 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.145822    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"]
Nov 03 11:30:00.149987 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.149964    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:00.160033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.160003    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"]
Nov 03 11:30:00.170359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.170328    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"]
Nov 03 11:30:00.194637 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.194605    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"]
Nov 03 11:30:00.199834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.199807    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"]
Nov 03 11:30:00.200014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.199898    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:00.204564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.204528    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"]
Nov 03 11:30:00.204724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.204654    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:00.210371 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.210349    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:00.222112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.222083    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"]
Nov 03 11:30:00.249293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.249266    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"]
Nov 03 11:30:00.256630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.256600    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"]
Nov 03 11:30:00.256868 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.256851    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"]
Nov 03 11:30:00.257032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.257005    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"]
Nov 03 11:30:00.259289 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.259266    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:00.263019 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.262902    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"]
Nov 03 11:30:00.263155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.263107    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:00.266246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.264458    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"]
Nov 03 11:30:00.266449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.266409    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7171db8f-769b-4dba-9e97-31b1a5ed949d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps\" (UID: \"7171db8f-769b-4dba-9e97-31b1a5ed949d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:00.266743 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.266719    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvln7\" (UniqueName: \"kubernetes.io/projected/e7e50e5e-7b9a-4a24-9def-949fed6aca68-kube-api-access-nvln7\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4\" (UID: \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:00.267233 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.267211    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcddg\" (UniqueName: \"kubernetes.io/projected/7171db8f-769b-4dba-9e97-31b1a5ed949d-kube-api-access-jcddg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps\" (UID: \"7171db8f-769b-4dba-9e97-31b1a5ed949d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:00.267446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.267428    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e50e5e-7b9a-4a24-9def-949fed6aca68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4\" (UID: \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:00.270742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.270719    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"]
Nov 03 11:30:00.270853 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.270824    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:00.272287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.272263    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"]
Nov 03 11:30:00.369272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369238    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471f345-99b5-4ca1-b2fb-1995db880fda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj\" (UID: \"9471f345-99b5-4ca1-b2fb-1995db880fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:00.369460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369293    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcddg\" (UniqueName: \"kubernetes.io/projected/7171db8f-769b-4dba-9e97-31b1a5ed949d-kube-api-access-jcddg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps\" (UID: \"7171db8f-769b-4dba-9e97-31b1a5ed949d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:00.369460 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369398    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kht\" (UniqueName: \"kubernetes.io/projected/78ff459d-34b6-440e-afe0-061e1d7860a9-kube-api-access-p8kht\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt\" (UID: \"78ff459d-34b6-440e-afe0-061e1d7860a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:00.369590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369459    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e50e5e-7b9a-4a24-9def-949fed6aca68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4\" (UID: \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:00.369590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369501    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b4527-5c0d-40c7-9e8b-03417efc5d9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895090-5rts6\" (UID: \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:00.369590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369531    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7171db8f-769b-4dba-9e97-31b1a5ed949d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps\" (UID: \"7171db8f-769b-4dba-9e97-31b1a5ed949d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:00.369590 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369551    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639d6260-0fdd-4698-83bc-61060581644f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq\" (UID: \"639d6260-0fdd-4698-83bc-61060581644f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:00.369827 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369673    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbjfg\" (UniqueName: \"kubernetes.io/projected/049a4cb6-866c-4d73-a260-7383b3fa0891-kube-api-access-vbjfg\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k\" (UID: \"049a4cb6-866c-4d73-a260-7383b3fa0891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:00.369869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369831    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s4gb\" (UniqueName: \"kubernetes.io/projected/799b4527-5c0d-40c7-9e8b-03417efc5d9c-kube-api-access-6s4gb\") pod \"maas-api-key-cleanup-29895090-5rts6\" (UID: \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:00.369931 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.369880    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdr5\" (UniqueName: \"kubernetes.io/projected/639d6260-0fdd-4698-83bc-61060581644f-kube-api-access-kjdr5\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq\" (UID: \"639d6260-0fdd-4698-83bc-61060581644f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:00.370032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.370012    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9njq\" (UniqueName: \"kubernetes.io/projected/babd4cd3-b063-475c-adc0-792eff654a76-kube-api-access-b9njq\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w\" (UID: \"babd4cd3-b063-475c-adc0-792eff654a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:00.370156 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.370141    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babd4cd3-b063-475c-adc0-792eff654a76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w\" (UID: \"babd4cd3-b063-475c-adc0-792eff654a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:00.370195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.370158    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7171db8f-769b-4dba-9e97-31b1a5ed949d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps\" (UID: \"7171db8f-769b-4dba-9e97-31b1a5ed949d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:00.370231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.370194    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049a4cb6-866c-4d73-a260-7383b3fa0891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k\" (UID: \"049a4cb6-866c-4d73-a260-7383b3fa0891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:00.370231 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.370227    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvln7\" (UniqueName: \"kubernetes.io/projected/e7e50e5e-7b9a-4a24-9def-949fed6aca68-kube-api-access-nvln7\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4\" (UID: \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:00.370299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.370255    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e50e5e-7b9a-4a24-9def-949fed6aca68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4\" (UID: \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:00.370299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.370288    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ff459d-34b6-440e-afe0-061e1d7860a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt\" (UID: \"78ff459d-34b6-440e-afe0-061e1d7860a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:00.370380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.370312    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxm2j\" (UniqueName: \"kubernetes.io/projected/9471f345-99b5-4ca1-b2fb-1995db880fda-kube-api-access-rxm2j\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj\" (UID: \"9471f345-99b5-4ca1-b2fb-1995db880fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:00.378571 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.378547    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvln7\" (UniqueName: \"kubernetes.io/projected/e7e50e5e-7b9a-4a24-9def-949fed6aca68-kube-api-access-nvln7\") pod \"maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4\" (UID: \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:00.378705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.378641    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcddg\" (UniqueName: \"kubernetes.io/projected/7171db8f-769b-4dba-9e97-31b1a5ed949d-kube-api-access-jcddg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps\" (UID: \"7171db8f-769b-4dba-9e97-31b1a5ed949d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:00.464630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.464589    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472326    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kht\" (UniqueName: \"kubernetes.io/projected/78ff459d-34b6-440e-afe0-061e1d7860a9-kube-api-access-p8kht\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt\" (UID: \"78ff459d-34b6-440e-afe0-061e1d7860a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472410    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b4527-5c0d-40c7-9e8b-03417efc5d9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895090-5rts6\" (UID: \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472457    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639d6260-0fdd-4698-83bc-61060581644f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq\" (UID: \"639d6260-0fdd-4698-83bc-61060581644f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472494    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjfg\" (UniqueName: \"kubernetes.io/projected/049a4cb6-866c-4d73-a260-7383b3fa0891-kube-api-access-vbjfg\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k\" (UID: \"049a4cb6-866c-4d73-a260-7383b3fa0891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472582    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s4gb\" (UniqueName: \"kubernetes.io/projected/799b4527-5c0d-40c7-9e8b-03417efc5d9c-kube-api-access-6s4gb\") pod \"maas-api-key-cleanup-29895090-5rts6\" (UID: \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472617    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdr5\" (UniqueName: \"kubernetes.io/projected/639d6260-0fdd-4698-83bc-61060581644f-kube-api-access-kjdr5\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq\" (UID: \"639d6260-0fdd-4698-83bc-61060581644f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472652    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9njq\" (UniqueName: \"kubernetes.io/projected/babd4cd3-b063-475c-adc0-792eff654a76-kube-api-access-b9njq\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w\" (UID: \"babd4cd3-b063-475c-adc0-792eff654a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472723    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babd4cd3-b063-475c-adc0-792eff654a76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w\" (UID: \"babd4cd3-b063-475c-adc0-792eff654a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472761    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049a4cb6-866c-4d73-a260-7383b3fa0891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k\" (UID: \"049a4cb6-866c-4d73-a260-7383b3fa0891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472825    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ff459d-34b6-440e-afe0-061e1d7860a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt\" (UID: \"78ff459d-34b6-440e-afe0-061e1d7860a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472854    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxm2j\" (UniqueName: \"kubernetes.io/projected/9471f345-99b5-4ca1-b2fb-1995db880fda-kube-api-access-rxm2j\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj\" (UID: \"9471f345-99b5-4ca1-b2fb-1995db880fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.472890    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471f345-99b5-4ca1-b2fb-1995db880fda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj\" (UID: \"9471f345-99b5-4ca1-b2fb-1995db880fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.473419    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639d6260-0fdd-4698-83bc-61060581644f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq\" (UID: \"639d6260-0fdd-4698-83bc-61060581644f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.473672    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471f345-99b5-4ca1-b2fb-1995db880fda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj\" (UID: \"9471f345-99b5-4ca1-b2fb-1995db880fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.474103    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b4527-5c0d-40c7-9e8b-03417efc5d9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895090-5rts6\" (UID: \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.474653    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.474795    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babd4cd3-b063-475c-adc0-792eff654a76-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w\" (UID: \"babd4cd3-b063-475c-adc0-792eff654a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:00.475687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.475630    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ff459d-34b6-440e-afe0-061e1d7860a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt\" (UID: \"78ff459d-34b6-440e-afe0-061e1d7860a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:00.477452 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.475983    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049a4cb6-866c-4d73-a260-7383b3fa0891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k\" (UID: \"049a4cb6-866c-4d73-a260-7383b3fa0891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:00.493091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.488880    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjfg\" (UniqueName: \"kubernetes.io/projected/049a4cb6-866c-4d73-a260-7383b3fa0891-kube-api-access-vbjfg\") pod \"maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k\" (UID: \"049a4cb6-866c-4d73-a260-7383b3fa0891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:00.516940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.514998    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdr5\" (UniqueName: \"kubernetes.io/projected/639d6260-0fdd-4698-83bc-61060581644f-kube-api-access-kjdr5\") pod \"maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq\" (UID: \"639d6260-0fdd-4698-83bc-61060581644f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:00.516940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.515626    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kht\" (UniqueName: \"kubernetes.io/projected/78ff459d-34b6-440e-afe0-061e1d7860a9-kube-api-access-p8kht\") pod \"maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt\" (UID: \"78ff459d-34b6-440e-afe0-061e1d7860a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:00.516940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.516595    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s4gb\" (UniqueName: \"kubernetes.io/projected/799b4527-5c0d-40c7-9e8b-03417efc5d9c-kube-api-access-6s4gb\") pod \"maas-api-key-cleanup-29895090-5rts6\" (UID: \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:00.521256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.519557    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9njq\" (UniqueName: \"kubernetes.io/projected/babd4cd3-b063-475c-adc0-792eff654a76-kube-api-access-b9njq\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w\" (UID: \"babd4cd3-b063-475c-adc0-792eff654a76\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:00.521256 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.520190    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxm2j\" (UniqueName: \"kubernetes.io/projected/9471f345-99b5-4ca1-b2fb-1995db880fda-kube-api-access-rxm2j\") pod \"maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj\" (UID: \"9471f345-99b5-4ca1-b2fb-1995db880fda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:00.540984 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.540231    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:00.567227 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.567191    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:00.568471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.568079    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:30:00.575958 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.575731    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:00.602809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.602578    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:00.603424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.603355    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:00.609363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.608903    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:00.682366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.675225    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-xks-ca\") pod \"b03af0f2-633e-48b4-ac32-f78749a60da3\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") "
Nov 03 11:30:00.682366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.675312    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-openshift-ca\") pod \"b03af0f2-633e-48b4-ac32-f78749a60da3\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") "
Nov 03 11:30:00.682366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.680806    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-openshift-ca" pod "b03af0f2-633e-48b4-ac32-f78749a60da3" (UID: "b03af0f2-633e-48b4-ac32-f78749a60da3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:00.686336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.686156    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03af0f2-633e-48b4-ac32-f78749a60da3-kube-api-access-22jlp" pod "b03af0f2-633e-48b4-ac32-f78749a60da3" (UID: "b03af0f2-633e-48b4-ac32-f78749a60da3"). InnerVolumeSpecName "kube-api-access-22jlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:00.687297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.675374    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03af0f2-633e-48b4-ac32-f78749a60da3-kube-api-access-22jlp\" (UniqueName: \"kubernetes.io/projected/b03af0f2-633e-48b4-ac32-f78749a60da3-kube-api-access-22jlp\") pod \"b03af0f2-633e-48b4-ac32-f78749a60da3\" (UID: \"b03af0f2-633e-48b4-ac32-f78749a60da3\") "
Nov 03 11:30:00.687297 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.687142    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-xks-ca" pod "b03af0f2-633e-48b4-ac32-f78749a60da3" (UID: "b03af0f2-633e-48b4-ac32-f78749a60da3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:00.688577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.688368    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:00.688577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.688399    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22jlp\" (UniqueName: \"kubernetes.io/projected/b03af0f2-633e-48b4-ac32-f78749a60da3-kube-api-access-22jlp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:00.688577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.688416    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03af0f2-633e-48b4-ac32-f78749a60da3-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:00.984569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:00.984532    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"]
Nov 03 11:30:00.985834 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:30:00.985806    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabd4cd3_b063_475c_adc0_792eff654a76.slice/crio-1075ba0fce8b90bbe5a7053a25047e58420ac87796f3ff6cf9cbbb6d2a663303 WatchSource:0}: Error finding container 1075ba0fce8b90bbe5a7053a25047e58420ac87796f3ff6cf9cbbb6d2a663303: Status 404 returned error can't find the container with id 1075ba0fce8b90bbe5a7053a25047e58420ac87796f3ff6cf9cbbb6d2a663303
Nov 03 11:30:01.027998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.027964    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"]
Nov 03 11:30:01.039964 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:30:01.038203    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639d6260_0fdd_4698_83bc_61060581644f.slice/crio-af7044996f80f17aa13e4f5828bd4ac9bf8200bd967e8955caebfd81e7b2c973 WatchSource:0}: Error finding container af7044996f80f17aa13e4f5828bd4ac9bf8200bd967e8955caebfd81e7b2c973: Status 404 returned error can't find the container with id af7044996f80f17aa13e4f5828bd4ac9bf8200bd967e8955caebfd81e7b2c973
Nov 03 11:30:01.043511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.043484    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"]
Nov 03 11:30:01.046818 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.046794    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"]
Nov 03 11:30:01.059407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.059388    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"]
Nov 03 11:30:01.064511 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.064486    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"]
Nov 03 11:30:01.089291 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.089262    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"]
Nov 03 11:30:01.115288 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.115264    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"]
Nov 03 11:30:01.184234 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:30:01.184174    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9471f345_99b5_4ca1_b2fb_1995db880fda.slice/crio-2b073b9d1cf9dfaf5c695829d4b9a2c149bd86bf3ed006a8a5c7a823a40f103c WatchSource:0}: Error finding container 2b073b9d1cf9dfaf5c695829d4b9a2c149bd86bf3ed006a8a5c7a823a40f103c: Status 404 returned error can't find the container with id 2b073b9d1cf9dfaf5c695829d4b9a2c149bd86bf3ed006a8a5c7a823a40f103c
Nov 03 11:30:01.286085 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:30:01.285959    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ff459d_34b6_440e_afe0_061e1d7860a9.slice/crio-1ac3814c17946f24e206f0a8e8c7b7008a2e0664cb5b26af9a35d13eb97df042 WatchSource:0}: Error finding container 1ac3814c17946f24e206f0a8e8c7b7008a2e0664cb5b26af9a35d13eb97df042: Status 404 returned error can't find the container with id 1ac3814c17946f24e206f0a8e8c7b7008a2e0664cb5b26af9a35d13eb97df042
Nov 03 11:30:01.412960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.412886    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w" event={"ID":"babd4cd3-b063-475c-adc0-792eff654a76","Type":"ContainerStarted","Data":"1075ba0fce8b90bbe5a7053a25047e58420ac87796f3ff6cf9cbbb6d2a663303"}
Nov 03 11:30:01.414684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.414665    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"
Nov 03 11:30:01.414805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.414664    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7" event={"ID":"b03af0f2-633e-48b4-ac32-f78749a60da3","Type":"ContainerDied","Data":"d947062c27e613656c510e632d0f6cd558984a44281527ece09b782c3581425e"}
Nov 03 11:30:01.414880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.414832    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d947062c27e613656c510e632d0f6cd558984a44281527ece09b782c3581425e"
Nov 03 11:30:01.416313 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.416289    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" event={"ID":"9471f345-99b5-4ca1-b2fb-1995db880fda","Type":"ContainerStarted","Data":"2b073b9d1cf9dfaf5c695829d4b9a2c149bd86bf3ed006a8a5c7a823a40f103c"}
Nov 03 11:30:01.417763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.417734    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" event={"ID":"7171db8f-769b-4dba-9e97-31b1a5ed949d","Type":"ContainerStarted","Data":"08ed580cf4df5ea2f1de3d6098bb69de6387a6383df12fb38f3b185a8cdde968"}
Nov 03 11:30:01.419145 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.419125    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" event={"ID":"639d6260-0fdd-4698-83bc-61060581644f","Type":"ContainerStarted","Data":"af7044996f80f17aa13e4f5828bd4ac9bf8200bd967e8955caebfd81e7b2c973"}
Nov 03 11:30:01.420877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.420826    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" event={"ID":"049a4cb6-866c-4d73-a260-7383b3fa0891","Type":"ContainerStarted","Data":"6b90c49f6c8956193b750d0cb1c2539a301b28a034b6763e3320bf0324618bf0"}
Nov 03 11:30:01.423952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.423931    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" event={"ID":"799b4527-5c0d-40c7-9e8b-03417efc5d9c","Type":"ContainerStarted","Data":"d4680ea2b12092dbb4c9e283268cae698ff3982bf547733900bbef8cd6417d44"}
Nov 03 11:30:01.425348 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.425329    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" event={"ID":"78ff459d-34b6-440e-afe0-061e1d7860a9","Type":"ContainerStarted","Data":"1ac3814c17946f24e206f0a8e8c7b7008a2e0664cb5b26af9a35d13eb97df042"}
Nov 03 11:30:01.426674 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:01.426653    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" event={"ID":"e7e50e5e-7b9a-4a24-9def-949fed6aca68","Type":"ContainerStarted","Data":"9db14ae26c9a218d974a90789326ea99cdb8297fb5f15c19ddf797e99b98c38d"}
Nov 03 11:30:02.436821 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.436789    5393 generic.go:356] "Generic (PLEG): container finished" podID="639d6260-0fdd-4698-83bc-61060581644f" containerID="8bef758781a821229b41cbd28fef90039ca9495f05a5f27a553f8f618aee32bd" exitCode=60
Nov 03 11:30:02.437346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.436883    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" event={"ID":"639d6260-0fdd-4698-83bc-61060581644f","Type":"ContainerDied","Data":"8bef758781a821229b41cbd28fef90039ca9495f05a5f27a553f8f618aee32bd"}
Nov 03 11:30:02.437346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.437241    5393 scope.go:122] "RemoveContainer" containerID="8bef758781a821229b41cbd28fef90039ca9495f05a5f27a553f8f618aee32bd"
Nov 03 11:30:02.439068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.439042    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" event={"ID":"049a4cb6-866c-4d73-a260-7383b3fa0891","Type":"ContainerStarted","Data":"9e020f00e35cff12162333010e968697ddc60eeaa59a01eb46f9847813586d7e"}
Nov 03 11:30:02.441325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.441181    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" event={"ID":"799b4527-5c0d-40c7-9e8b-03417efc5d9c","Type":"ContainerStarted","Data":"d5f29d0860c2b5018c6188dfb73163f2b0ca1460c32d13a74b3cea88d1a7bb87"}
Nov 03 11:30:02.442965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.442941    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" event={"ID":"78ff459d-34b6-440e-afe0-061e1d7860a9","Type":"ContainerStarted","Data":"6a1e748221c2de979435e264bd42c94ac00a2bf03f6fec8452f0958c354847a4"}
Nov 03 11:30:02.444663 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.444637    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" event={"ID":"e7e50e5e-7b9a-4a24-9def-949fed6aca68","Type":"ContainerStarted","Data":"b489becf8b6a29ad07c1d90e5185f8bfb4986887350119be634a95935ff514b1"}
Nov 03 11:30:02.446330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.446305    5393 generic.go:356] "Generic (PLEG): container finished" podID="babd4cd3-b063-475c-adc0-792eff654a76" containerID="6edbd8fd93c15de5f96ac3e388c85e792eaeeadeb36da8583ad4632f2c8460ba" exitCode=60
Nov 03 11:30:02.446454 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.446376    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w" event={"ID":"babd4cd3-b063-475c-adc0-792eff654a76","Type":"ContainerDied","Data":"6edbd8fd93c15de5f96ac3e388c85e792eaeeadeb36da8583ad4632f2c8460ba"}
Nov 03 11:30:02.446585 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.446568    5393 scope.go:122] "RemoveContainer" containerID="6edbd8fd93c15de5f96ac3e388c85e792eaeeadeb36da8583ad4632f2c8460ba"
Nov 03 11:30:02.448011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.447993    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" event={"ID":"9471f345-99b5-4ca1-b2fb-1995db880fda","Type":"ContainerStarted","Data":"cd0a4b2e3cd11ff912acf770df24ae8f8fa5726de267cdb9651582c967e905c6"}
Nov 03 11:30:02.449636 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.449611    5393 generic.go:356] "Generic (PLEG): container finished" podID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerID="7f33144879cb81a7ed10aaf39dda446e89407adbbcbf0574deb02d26ed675b03" exitCode=60
Nov 03 11:30:02.449727 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.449686    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" event={"ID":"7171db8f-769b-4dba-9e97-31b1a5ed949d","Type":"ContainerDied","Data":"7f33144879cb81a7ed10aaf39dda446e89407adbbcbf0574deb02d26ed675b03"}
Nov 03 11:30:02.449852 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.449836    5393 scope.go:122] "RemoveContainer" containerID="7f33144879cb81a7ed10aaf39dda446e89407adbbcbf0574deb02d26ed675b03"
Nov 03 11:30:02.469575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.469512    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" podStartSLOduration=2.469494075 podStartE2EDuration="2.469494075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:02.467130601 +0000 UTC m=+3646.371985042" watchObservedRunningTime="2026-11-03 11:30:02.469494075 +0000 UTC m=+3646.374348517"
Nov 03 11:30:02.527061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.526949    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" podStartSLOduration=2.526934028 podStartE2EDuration="2.526934028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:02.523360178 +0000 UTC m=+3646.428214619" watchObservedRunningTime="2026-11-03 11:30:02.526934028 +0000 UTC m=+3646.431788463"
Nov 03 11:30:02.539677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.539635    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" podStartSLOduration=2.539621133 podStartE2EDuration="2.539621133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:02.537893527 +0000 UTC m=+3646.442747968" watchObservedRunningTime="2026-11-03 11:30:02.539621133 +0000 UTC m=+3646.444475574"
Nov 03 11:30:02.557177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.557130    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" podStartSLOduration=2.557106929 podStartE2EDuration="2.557106929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:02.554112843 +0000 UTC m=+3646.458967285" watchObservedRunningTime="2026-11-03 11:30:02.557106929 +0000 UTC m=+3646.461961394"
Nov 03 11:30:02.571125 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:02.571074    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" podStartSLOduration=2.571059329 podStartE2EDuration="2.571059329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:02.568989202 +0000 UTC m=+3646.473843646" watchObservedRunningTime="2026-11-03 11:30:02.571059329 +0000 UTC m=+3646.475913770"
Nov 03 11:30:03.458806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.458765    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w" event={"ID":"babd4cd3-b063-475c-adc0-792eff654a76","Type":"ContainerStarted","Data":"629ac3db9a175de23020680163d8b0ee603f45b785e89199426d4b5d1b7f0868"}
Nov 03 11:30:03.460763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.460734    5393 generic.go:356] "Generic (PLEG): container finished" podID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerID="cd0a4b2e3cd11ff912acf770df24ae8f8fa5726de267cdb9651582c967e905c6" exitCode=60
Nov 03 11:30:03.460963 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.460825    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" event={"ID":"9471f345-99b5-4ca1-b2fb-1995db880fda","Type":"ContainerDied","Data":"cd0a4b2e3cd11ff912acf770df24ae8f8fa5726de267cdb9651582c967e905c6"}
Nov 03 11:30:03.461131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.461111    5393 scope.go:122] "RemoveContainer" containerID="cd0a4b2e3cd11ff912acf770df24ae8f8fa5726de267cdb9651582c967e905c6"
Nov 03 11:30:03.463308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.463281    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" event={"ID":"7171db8f-769b-4dba-9e97-31b1a5ed949d","Type":"ContainerStarted","Data":"db9cae4efe218c8fa92192b6e2b055ff6d82710757258140d8324764ab9d8d11"}
Nov 03 11:30:03.465488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.465463    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" event={"ID":"639d6260-0fdd-4698-83bc-61060581644f","Type":"ContainerStarted","Data":"6c85a2a094e5d7ce2579a9771cd50caf7eb4f99d2ffc5df151d0d8559c3fe1b4"}
Nov 03 11:30:03.467131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.467108    5393 generic.go:356] "Generic (PLEG): container finished" podID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerID="9e020f00e35cff12162333010e968697ddc60eeaa59a01eb46f9847813586d7e" exitCode=60
Nov 03 11:30:03.467218 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.467191    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" event={"ID":"049a4cb6-866c-4d73-a260-7383b3fa0891","Type":"ContainerDied","Data":"9e020f00e35cff12162333010e968697ddc60eeaa59a01eb46f9847813586d7e"}
Nov 03 11:30:03.467440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.467414    5393 scope.go:122] "RemoveContainer" containerID="9e020f00e35cff12162333010e968697ddc60eeaa59a01eb46f9847813586d7e"
Nov 03 11:30:03.468833 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.468814    5393 generic.go:356] "Generic (PLEG): container finished" podID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerID="d5f29d0860c2b5018c6188dfb73163f2b0ca1460c32d13a74b3cea88d1a7bb87" exitCode=60
Nov 03 11:30:03.468956 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.468875    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" event={"ID":"799b4527-5c0d-40c7-9e8b-03417efc5d9c","Type":"ContainerDied","Data":"d5f29d0860c2b5018c6188dfb73163f2b0ca1460c32d13a74b3cea88d1a7bb87"}
Nov 03 11:30:03.469106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.469083    5393 scope.go:122] "RemoveContainer" containerID="d5f29d0860c2b5018c6188dfb73163f2b0ca1460c32d13a74b3cea88d1a7bb87"
Nov 03 11:30:03.471323 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.471305    5393 generic.go:356] "Generic (PLEG): container finished" podID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerID="6a1e748221c2de979435e264bd42c94ac00a2bf03f6fec8452f0958c354847a4" exitCode=60
Nov 03 11:30:03.471411 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.471378    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" event={"ID":"78ff459d-34b6-440e-afe0-061e1d7860a9","Type":"ContainerDied","Data":"6a1e748221c2de979435e264bd42c94ac00a2bf03f6fec8452f0958c354847a4"}
Nov 03 11:30:03.471603 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.471588    5393 scope.go:122] "RemoveContainer" containerID="6a1e748221c2de979435e264bd42c94ac00a2bf03f6fec8452f0958c354847a4"
Nov 03 11:30:03.473250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.473225    5393 generic.go:356] "Generic (PLEG): container finished" podID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerID="b489becf8b6a29ad07c1d90e5185f8bfb4986887350119be634a95935ff514b1" exitCode=60
Nov 03 11:30:03.473358 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.473289    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" event={"ID":"e7e50e5e-7b9a-4a24-9def-949fed6aca68","Type":"ContainerDied","Data":"b489becf8b6a29ad07c1d90e5185f8bfb4986887350119be634a95935ff514b1"}
Nov 03 11:30:03.473607 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.473493    5393 scope.go:122] "RemoveContainer" containerID="b489becf8b6a29ad07c1d90e5185f8bfb4986887350119be634a95935ff514b1"
Nov 03 11:30:03.482710 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.482657    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w" podStartSLOduration=3.482640664 podStartE2EDuration="3.482640664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:03.479263398 +0000 UTC m=+3647.384117839" watchObservedRunningTime="2026-11-03 11:30:03.482640664 +0000 UTC m=+3647.387495109"
Nov 03 11:30:03.561254 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.561198    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" podStartSLOduration=3.561183292 podStartE2EDuration="3.561183292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:03.557653682 +0000 UTC m=+3647.462508125" watchObservedRunningTime="2026-11-03 11:30:03.561183292 +0000 UTC m=+3647.466037733"
Nov 03 11:30:03.583646 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.581632    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" podStartSLOduration=3.58161846 podStartE2EDuration="3.58161846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:03.579423043 +0000 UTC m=+3647.484277484" watchObservedRunningTime="2026-11-03 11:30:03.58161846 +0000 UTC m=+3647.486472904"
Nov 03 11:30:03.854154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:03.854112    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"]
Nov 03 11:30:04.485341 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.485305    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" event={"ID":"799b4527-5c0d-40c7-9e8b-03417efc5d9c","Type":"ContainerStarted","Data":"dc8751af4536494501df48266efd193e85442685a86700340228418dd6b65578"}
Nov 03 11:30:04.487576 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.487548    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" event={"ID":"78ff459d-34b6-440e-afe0-061e1d7860a9","Type":"ContainerStarted","Data":"6334c5e82f920c22420071540abb7755f7a161347d62b19516954c8493941c51"}
Nov 03 11:30:04.489999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.489967    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" event={"ID":"e7e50e5e-7b9a-4a24-9def-949fed6aca68","Type":"ContainerStarted","Data":"f7fe0fe8b781f0c7bd9b1b12829c262ec840b0a8599711b51ffa7a6de6e97224"}
Nov 03 11:30:04.492424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.492401    5393 generic.go:356] "Generic (PLEG): container finished" podID="babd4cd3-b063-475c-adc0-792eff654a76" containerID="629ac3db9a175de23020680163d8b0ee603f45b785e89199426d4b5d1b7f0868" exitCode=60
Nov 03 11:30:04.492567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.492482    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w" event={"ID":"babd4cd3-b063-475c-adc0-792eff654a76","Type":"ContainerDied","Data":"629ac3db9a175de23020680163d8b0ee603f45b785e89199426d4b5d1b7f0868"}
Nov 03 11:30:04.492567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.492536    5393 scope.go:122] "RemoveContainer" containerID="6edbd8fd93c15de5f96ac3e388c85e792eaeeadeb36da8583ad4632f2c8460ba"
Nov 03 11:30:04.492729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.492708    5393 scope.go:122] "RemoveContainer" containerID="629ac3db9a175de23020680163d8b0ee603f45b785e89199426d4b5d1b7f0868"
Nov 03 11:30:04.493059 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:04.493034    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w_odh-ai-gateway-infra(babd4cd3-b063-475c-adc0-792eff654a76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w" podUID="babd4cd3-b063-475c-adc0-792eff654a76"
Nov 03 11:30:04.494872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.494846    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" event={"ID":"9471f345-99b5-4ca1-b2fb-1995db880fda","Type":"ContainerStarted","Data":"d07bc8049ad1731e24ff663f6e45684b97f77f37eaee73b4f32360e2e26d59d3"}
Nov 03 11:30:04.496161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.496128    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main" containerID="cri-o://9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11" gracePeriod=45
Nov 03 11:30:04.497279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.497254    5393 generic.go:356] "Generic (PLEG): container finished" podID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerID="db9cae4efe218c8fa92192b6e2b055ff6d82710757258140d8324764ab9d8d11" exitCode=60
Nov 03 11:30:04.497436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.497276    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" event={"ID":"7171db8f-769b-4dba-9e97-31b1a5ed949d","Type":"ContainerDied","Data":"db9cae4efe218c8fa92192b6e2b055ff6d82710757258140d8324764ab9d8d11"}
Nov 03 11:30:04.497569 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.497552    5393 scope.go:122] "RemoveContainer" containerID="db9cae4efe218c8fa92192b6e2b055ff6d82710757258140d8324764ab9d8d11"
Nov 03 11:30:04.497982 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:04.497910    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps_odh-ai-gateway-infra(7171db8f-769b-4dba-9e97-31b1a5ed949d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d"
Nov 03 11:30:04.500085 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.500063    5393 generic.go:356] "Generic (PLEG): container finished" podID="639d6260-0fdd-4698-83bc-61060581644f" containerID="6c85a2a094e5d7ce2579a9771cd50caf7eb4f99d2ffc5df151d0d8559c3fe1b4" exitCode=60
Nov 03 11:30:04.500193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.500143    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" event={"ID":"639d6260-0fdd-4698-83bc-61060581644f","Type":"ContainerDied","Data":"6c85a2a094e5d7ce2579a9771cd50caf7eb4f99d2ffc5df151d0d8559c3fe1b4"}
Nov 03 11:30:04.500330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.500308    5393 scope.go:122] "RemoveContainer" containerID="6c85a2a094e5d7ce2579a9771cd50caf7eb4f99d2ffc5df151d0d8559c3fe1b4"
Nov 03 11:30:04.500592 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:04.500570    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq_odh-ai-gateway-infra(639d6260-0fdd-4698-83bc-61060581644f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" podUID="639d6260-0fdd-4698-83bc-61060581644f"
Nov 03 11:30:04.503525 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.503505    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" event={"ID":"049a4cb6-866c-4d73-a260-7383b3fa0891","Type":"ContainerStarted","Data":"d008d4d5d16d14ac987bd25f23e59fdc3280c7f7279117d846354be921982698"}
Nov 03 11:30:04.512843 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.512826    5393 scope.go:122] "RemoveContainer" containerID="7f33144879cb81a7ed10aaf39dda446e89407adbbcbf0574deb02d26ed675b03"
Nov 03 11:30:04.613850 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.613825    5393 scope.go:122] "RemoveContainer" containerID="8bef758781a821229b41cbd28fef90039ca9495f05a5f27a553f8f618aee32bd"
Nov 03 11:30:04.777096 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.777068    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 11:30:04.862764 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.862730    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-model-cache\") pod \"e2977665-bb0a-4a29-a5a1-5bf515156271\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") "
Nov 03 11:30:04.862986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.862793    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2977665-bb0a-4a29-a5a1-5bf515156271-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2977665-bb0a-4a29-a5a1-5bf515156271-tls-certs\") pod \"e2977665-bb0a-4a29-a5a1-5bf515156271\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") "
Nov 03 11:30:04.862986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.862850    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-tmp-dir\") pod \"e2977665-bb0a-4a29-a5a1-5bf515156271\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") "
Nov 03 11:30:04.862986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.862904    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-dshm\") pod \"e2977665-bb0a-4a29-a5a1-5bf515156271\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") "
Nov 03 11:30:04.862986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.862955    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2977665-bb0a-4a29-a5a1-5bf515156271-kube-api-access-b2b85\" (UniqueName: \"kubernetes.io/projected/e2977665-bb0a-4a29-a5a1-5bf515156271-kube-api-access-b2b85\") pod \"e2977665-bb0a-4a29-a5a1-5bf515156271\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") "
Nov 03 11:30:04.863244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.863021    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-model-cache" pod "e2977665-bb0a-4a29-a5a1-5bf515156271" (UID: "e2977665-bb0a-4a29-a5a1-5bf515156271"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.863244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.863029    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-home\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-home\") pod \"e2977665-bb0a-4a29-a5a1-5bf515156271\" (UID: \"e2977665-bb0a-4a29-a5a1-5bf515156271\") "
Nov 03 11:30:04.863244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.863207    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-tmp-dir" pod "e2977665-bb0a-4a29-a5a1-5bf515156271" (UID: "e2977665-bb0a-4a29-a5a1-5bf515156271"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.863415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.863244    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-home" pod "e2977665-bb0a-4a29-a5a1-5bf515156271" (UID: "e2977665-bb0a-4a29-a5a1-5bf515156271"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.863739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.863702    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:04.863739 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.863733    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:04.864015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.863747    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:04.881888 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.881474    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-dshm" pod "e2977665-bb0a-4a29-a5a1-5bf515156271" (UID: "e2977665-bb0a-4a29-a5a1-5bf515156271"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:04.881888 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.881681    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2977665-bb0a-4a29-a5a1-5bf515156271-tls-certs" pod "e2977665-bb0a-4a29-a5a1-5bf515156271" (UID: "e2977665-bb0a-4a29-a5a1-5bf515156271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:04.882094 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.881996    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2977665-bb0a-4a29-a5a1-5bf515156271-kube-api-access-b2b85" pod "e2977665-bb0a-4a29-a5a1-5bf515156271" (UID: "e2977665-bb0a-4a29-a5a1-5bf515156271"). InnerVolumeSpecName "kube-api-access-b2b85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:04.964795 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.964763    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2977665-bb0a-4a29-a5a1-5bf515156271-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:04.964795 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.964796    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2977665-bb0a-4a29-a5a1-5bf515156271-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:04.965021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:04.964806    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2b85\" (UniqueName: \"kubernetes.io/projected/e2977665-bb0a-4a29-a5a1-5bf515156271-kube-api-access-b2b85\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:05.205470 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.205424    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"]
Nov 03 11:30:05.525016 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.524979    5393 scope.go:122] "RemoveContainer" containerID="629ac3db9a175de23020680163d8b0ee603f45b785e89199426d4b5d1b7f0868"
Nov 03 11:30:05.526428 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.526394    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w_odh-ai-gateway-infra(babd4cd3-b063-475c-adc0-792eff654a76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w" podUID="babd4cd3-b063-475c-adc0-792eff654a76"
Nov 03 11:30:05.552192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.552140    5393 generic.go:356] "Generic (PLEG): container finished" podID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerID="d07bc8049ad1731e24ff663f6e45684b97f77f37eaee73b4f32360e2e26d59d3" exitCode=60
Nov 03 11:30:05.552391 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.552286    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" event={"ID":"9471f345-99b5-4ca1-b2fb-1995db880fda","Type":"ContainerDied","Data":"d07bc8049ad1731e24ff663f6e45684b97f77f37eaee73b4f32360e2e26d59d3"}
Nov 03 11:30:05.552489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.552410    5393 scope.go:122] "RemoveContainer" containerID="cd0a4b2e3cd11ff912acf770df24ae8f8fa5726de267cdb9651582c967e905c6"
Nov 03 11:30:05.553235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.553211    5393 scope.go:122] "RemoveContainer" containerID="d07bc8049ad1731e24ff663f6e45684b97f77f37eaee73b4f32360e2e26d59d3"
Nov 03 11:30:05.553676 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.553648    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj_odh-ai-gateway-infra(9471f345-99b5-4ca1-b2fb-1995db880fda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda"
Nov 03 11:30:05.558673 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.558649    5393 scope.go:122] "RemoveContainer" containerID="db9cae4efe218c8fa92192b6e2b055ff6d82710757258140d8324764ab9d8d11"
Nov 03 11:30:05.559098 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.559075    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps_odh-ai-gateway-infra(7171db8f-769b-4dba-9e97-31b1a5ed949d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d"
Nov 03 11:30:05.561428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.561404    5393 scope.go:122] "RemoveContainer" containerID="6c85a2a094e5d7ce2579a9771cd50caf7eb4f99d2ffc5df151d0d8559c3fe1b4"
Nov 03 11:30:05.561965 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.561943    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq_odh-ai-gateway-infra(639d6260-0fdd-4698-83bc-61060581644f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" podUID="639d6260-0fdd-4698-83bc-61060581644f"
Nov 03 11:30:05.569320 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.569289    5393 generic.go:356] "Generic (PLEG): container finished" podID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerID="d008d4d5d16d14ac987bd25f23e59fdc3280c7f7279117d846354be921982698" exitCode=60
Nov 03 11:30:05.569494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.569472    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" event={"ID":"049a4cb6-866c-4d73-a260-7383b3fa0891","Type":"ContainerDied","Data":"d008d4d5d16d14ac987bd25f23e59fdc3280c7f7279117d846354be921982698"}
Nov 03 11:30:05.570312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.570293    5393 scope.go:122] "RemoveContainer" containerID="d008d4d5d16d14ac987bd25f23e59fdc3280c7f7279117d846354be921982698"
Nov 03 11:30:05.570737 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.570705    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k_odh-ai-gateway-infra(049a4cb6-866c-4d73-a260-7383b3fa0891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891"
Nov 03 11:30:05.571591 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.571566    5393 scope.go:122] "RemoveContainer" containerID="9e020f00e35cff12162333010e968697ddc60eeaa59a01eb46f9847813586d7e"
Nov 03 11:30:05.572331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.572307    5393 generic.go:356] "Generic (PLEG): container finished" podID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerID="9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11" exitCode=0
Nov 03 11:30:05.572444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.572421    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" event={"ID":"e2977665-bb0a-4a29-a5a1-5bf515156271","Type":"ContainerDied","Data":"9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11"}
Nov 03 11:30:05.572518 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.572464    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv" event={"ID":"e2977665-bb0a-4a29-a5a1-5bf515156271","Type":"ContainerDied","Data":"8746b99f3822868ef646e89f9258c2424be73c9af76d9366e1cd3bb01e02aeac"}
Nov 03 11:30:05.572583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.572430    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"
Nov 03 11:30:05.575650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.575627    5393 generic.go:356] "Generic (PLEG): container finished" podID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerID="dc8751af4536494501df48266efd193e85442685a86700340228418dd6b65578" exitCode=60
Nov 03 11:30:05.575748 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.575689    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" event={"ID":"799b4527-5c0d-40c7-9e8b-03417efc5d9c","Type":"ContainerDied","Data":"dc8751af4536494501df48266efd193e85442685a86700340228418dd6b65578"}
Nov 03 11:30:05.576083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.576069    5393 scope.go:122] "RemoveContainer" containerID="dc8751af4536494501df48266efd193e85442685a86700340228418dd6b65578"
Nov 03 11:30:05.576297 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.576280    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895090-5rts6_odh-ai-gateway-infra(799b4527-5c0d-40c7-9e8b-03417efc5d9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c"
Nov 03 11:30:05.579122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.579097    5393 generic.go:356] "Generic (PLEG): container finished" podID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerID="6334c5e82f920c22420071540abb7755f7a161347d62b19516954c8493941c51" exitCode=60
Nov 03 11:30:05.579237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.579153    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" event={"ID":"78ff459d-34b6-440e-afe0-061e1d7860a9","Type":"ContainerDied","Data":"6334c5e82f920c22420071540abb7755f7a161347d62b19516954c8493941c51"}
Nov 03 11:30:05.579458 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.579441    5393 scope.go:122] "RemoveContainer" containerID="6334c5e82f920c22420071540abb7755f7a161347d62b19516954c8493941c51"
Nov 03 11:30:05.579664 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.579645    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt_odh-ai-gateway-infra(78ff459d-34b6-440e-afe0-061e1d7860a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9"
Nov 03 11:30:05.584652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.584613    5393 generic.go:356] "Generic (PLEG): container finished" podID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerID="f7fe0fe8b781f0c7bd9b1b12829c262ec840b0a8599711b51ffa7a6de6e97224" exitCode=60
Nov 03 11:30:05.584795 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.584713    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" event={"ID":"e7e50e5e-7b9a-4a24-9def-949fed6aca68","Type":"ContainerDied","Data":"f7fe0fe8b781f0c7bd9b1b12829c262ec840b0a8599711b51ffa7a6de6e97224"}
Nov 03 11:30:05.585083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.585066    5393 scope.go:122] "RemoveContainer" containerID="f7fe0fe8b781f0c7bd9b1b12829c262ec840b0a8599711b51ffa7a6de6e97224"
Nov 03 11:30:05.585307 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.585286    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4_odh-ai-gateway-infra(e7e50e5e-7b9a-4a24-9def-949fed6aca68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68"
Nov 03 11:30:05.588817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.588800    5393 scope.go:122] "RemoveContainer" containerID="9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11"
Nov 03 11:30:05.599753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.599730    5393 scope.go:122] "RemoveContainer" containerID="9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11"
Nov 03 11:30:05.600254 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:05.600230    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11\": container with ID starting with 9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11 not found: ID does not exist" containerID="9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11"
Nov 03 11:30:05.600334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.600263    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11"} err="failed to get container status \"9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11\": rpc error: code = NotFound desc = could not find container \"9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11\": container with ID starting with 9ed143aab18296bc9378ccd0e1e0f2835654885e983a1bffed934bdabaa92d11 not found: ID does not exist"
Nov 03 11:30:05.600334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.600280    5393 scope.go:122] "RemoveContainer" containerID="d5f29d0860c2b5018c6188dfb73163f2b0ca1460c32d13a74b3cea88d1a7bb87"
Nov 03 11:30:05.615155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.615132    5393 scope.go:122] "RemoveContainer" containerID="6a1e748221c2de979435e264bd42c94ac00a2bf03f6fec8452f0958c354847a4"
Nov 03 11:30:05.631155 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.631128    5393 scope.go:122] "RemoveContainer" containerID="b489becf8b6a29ad07c1d90e5185f8bfb4986887350119be634a95935ff514b1"
Nov 03 11:30:05.659029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.658993    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"]
Nov 03 11:30:05.662416 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:05.662390    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d5d3c3/distinct-2-w4-d5d3c3-kserve-67db7fb86-xstrv"]
Nov 03 11:30:06.233933 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:06.233895    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" path="/var/lib/kubelet/pods/e2977665-bb0a-4a29-a5a1-5bf515156271/volumes"
Nov 03 11:30:06.599280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:06.599252    5393 scope.go:122] "RemoveContainer" containerID="dc8751af4536494501df48266efd193e85442685a86700340228418dd6b65578"
Nov 03 11:30:06.599739 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:06.599583    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895090-5rts6_odh-ai-gateway-infra(799b4527-5c0d-40c7-9e8b-03417efc5d9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c"
Nov 03 11:30:06.601252 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:06.601226    5393 scope.go:122] "RemoveContainer" containerID="6334c5e82f920c22420071540abb7755f7a161347d62b19516954c8493941c51"
Nov 03 11:30:06.601549 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:06.601523    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt_odh-ai-gateway-infra(78ff459d-34b6-440e-afe0-061e1d7860a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9"
Nov 03 11:30:06.603130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:06.603109    5393 scope.go:122] "RemoveContainer" containerID="f7fe0fe8b781f0c7bd9b1b12829c262ec840b0a8599711b51ffa7a6de6e97224"
Nov 03 11:30:06.603391 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:06.603364    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4_odh-ai-gateway-infra(e7e50e5e-7b9a-4a24-9def-949fed6aca68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68"
Nov 03 11:30:06.604865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:06.604846    5393 scope.go:122] "RemoveContainer" containerID="d07bc8049ad1731e24ff663f6e45684b97f77f37eaee73b4f32360e2e26d59d3"
Nov 03 11:30:06.605136 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:06.605110    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj_odh-ai-gateway-infra(9471f345-99b5-4ca1-b2fb-1995db880fda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda"
Nov 03 11:30:06.606683 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:06.606667    5393 scope.go:122] "RemoveContainer" containerID="d008d4d5d16d14ac987bd25f23e59fdc3280c7f7279117d846354be921982698"
Nov 03 11:30:06.606897 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:06.606875    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k_odh-ai-gateway-infra(049a4cb6-866c-4d73-a260-7383b3fa0891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891"
Nov 03 11:30:13.197560 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:13.197512    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:30:15.223952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:15.223888    5393 scope.go:122] "RemoveContainer" containerID="dc8751af4536494501df48266efd193e85442685a86700340228418dd6b65578"
Nov 03 11:30:15.452087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:15.452050    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"]
Nov 03 11:30:15.758945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:15.709186    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" event={"ID":"799b4527-5c0d-40c7-9e8b-03417efc5d9c","Type":"ContainerStarted","Data":"799478c24934fdb772523c0be12a5e08d1aa5909c6796a374f1bc8d117096e40"}
Nov 03 11:30:15.758945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:15.725969    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"]
Nov 03 11:30:15.840196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:15.840122    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"]
Nov 03 11:30:15.841952 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:15.841911    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:15.847455 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:15.847399    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d5d3c3-rnv6c"]
Nov 03 11:30:16.013024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.011894    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babd4cd3-b063-475c-adc0-792eff654a76-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babd4cd3-b063-475c-adc0-792eff654a76-maas-api-service-ca\") pod \"babd4cd3-b063-475c-adc0-792eff654a76\" (UID: \"babd4cd3-b063-475c-adc0-792eff654a76\") "
Nov 03 11:30:16.013024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.012035    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babd4cd3-b063-475c-adc0-792eff654a76-kube-api-access-b9njq\" (UniqueName: \"kubernetes.io/projected/babd4cd3-b063-475c-adc0-792eff654a76-kube-api-access-b9njq\") pod \"babd4cd3-b063-475c-adc0-792eff654a76\" (UID: \"babd4cd3-b063-475c-adc0-792eff654a76\") "
Nov 03 11:30:16.021943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.015324    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babd4cd3-b063-475c-adc0-792eff654a76-maas-api-service-ca" pod "babd4cd3-b063-475c-adc0-792eff654a76" (UID: "babd4cd3-b063-475c-adc0-792eff654a76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:16.053028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.033124    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babd4cd3-b063-475c-adc0-792eff654a76-kube-api-access-b9njq" pod "babd4cd3-b063-475c-adc0-792eff654a76" (UID: "babd4cd3-b063-475c-adc0-792eff654a76"). InnerVolumeSpecName "kube-api-access-b9njq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:16.116656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.116618    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babd4cd3-b063-475c-adc0-792eff654a76-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:16.116947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.116906    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9njq\" (UniqueName: \"kubernetes.io/projected/babd4cd3-b063-475c-adc0-792eff654a76-kube-api-access-b9njq\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:16.228103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.227734    5393 scope.go:122] "RemoveContainer" containerID="6c85a2a094e5d7ce2579a9771cd50caf7eb4f99d2ffc5df151d0d8559c3fe1b4"
Nov 03 11:30:16.233871 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.233846    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506a91f8-4a09-4bb0-99e7-d351273500d9" path="/var/lib/kubelet/pods/506a91f8-4a09-4bb0-99e7-d351273500d9/volumes"
Nov 03 11:30:16.718311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.718266    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" event={"ID":"639d6260-0fdd-4698-83bc-61060581644f","Type":"ContainerStarted","Data":"c0c8a7a0a46fe1ef72166a0e9f16948e7085e0131ccf712fcbd977bf15f333e9"}
Nov 03 11:30:16.720358 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.720333    5393 generic.go:356] "Generic (PLEG): container finished" podID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerID="799478c24934fdb772523c0be12a5e08d1aa5909c6796a374f1bc8d117096e40" exitCode=60
Nov 03 11:30:16.720535 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.720399    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" event={"ID":"799b4527-5c0d-40c7-9e8b-03417efc5d9c","Type":"ContainerDied","Data":"799478c24934fdb772523c0be12a5e08d1aa5909c6796a374f1bc8d117096e40"}
Nov 03 11:30:16.720535 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.720451    5393 scope.go:122] "RemoveContainer" containerID="dc8751af4536494501df48266efd193e85442685a86700340228418dd6b65578"
Nov 03 11:30:16.720849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.720827    5393 scope.go:122] "RemoveContainer" containerID="799478c24934fdb772523c0be12a5e08d1aa5909c6796a374f1bc8d117096e40"
Nov 03 11:30:16.721245 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:16.721217    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895090-5rts6_odh-ai-gateway-infra(799b4527-5c0d-40c7-9e8b-03417efc5d9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c"
Nov 03 11:30:16.722185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.722163    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w" event={"ID":"babd4cd3-b063-475c-adc0-792eff654a76","Type":"ContainerDied","Data":"1075ba0fce8b90bbe5a7053a25047e58420ac87796f3ff6cf9cbbb6d2a663303"}
Nov 03 11:30:16.722185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.722176    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"
Nov 03 11:30:16.735483 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.735465    5393 scope.go:122] "RemoveContainer" containerID="629ac3db9a175de23020680163d8b0ee603f45b785e89199426d4b5d1b7f0868"
Nov 03 11:30:16.748619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.748597    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"]
Nov 03 11:30:16.752028 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.752002    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5d3c3-29895090-5tv8w"]
Nov 03 11:30:16.786251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:16.786170    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"]
Nov 03 11:30:17.223583 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.223548    5393 scope.go:122] "RemoveContainer" containerID="db9cae4efe218c8fa92192b6e2b055ff6d82710757258140d8324764ab9d8d11"
Nov 03 11:30:17.223732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.223617    5393 scope.go:122] "RemoveContainer" containerID="6334c5e82f920c22420071540abb7755f7a161347d62b19516954c8493941c51"
Nov 03 11:30:17.732932 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.732877    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" event={"ID":"7171db8f-769b-4dba-9e97-31b1a5ed949d","Type":"ContainerStarted","Data":"caae155eac956535c6877d04fdecb2446f7cb59e8aa267c60241900df651596c"}
Nov 03 11:30:17.734989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.734957    5393 generic.go:356] "Generic (PLEG): container finished" podID="639d6260-0fdd-4698-83bc-61060581644f" containerID="c0c8a7a0a46fe1ef72166a0e9f16948e7085e0131ccf712fcbd977bf15f333e9" exitCode=60
Nov 03 11:30:17.735136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.735024    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" event={"ID":"639d6260-0fdd-4698-83bc-61060581644f","Type":"ContainerDied","Data":"c0c8a7a0a46fe1ef72166a0e9f16948e7085e0131ccf712fcbd977bf15f333e9"}
Nov 03 11:30:17.735136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.735075    5393 scope.go:122] "RemoveContainer" containerID="6c85a2a094e5d7ce2579a9771cd50caf7eb4f99d2ffc5df151d0d8559c3fe1b4"
Nov 03 11:30:17.735378 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.735365    5393 scope.go:122] "RemoveContainer" containerID="c0c8a7a0a46fe1ef72166a0e9f16948e7085e0131ccf712fcbd977bf15f333e9"
Nov 03 11:30:17.735692 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:17.735670    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq_odh-ai-gateway-infra(639d6260-0fdd-4698-83bc-61060581644f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" podUID="639d6260-0fdd-4698-83bc-61060581644f"
Nov 03 11:30:17.739066 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.739037    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" event={"ID":"78ff459d-34b6-440e-afe0-061e1d7860a9","Type":"ContainerStarted","Data":"16d0845f74f84c5925bf6753809b1566f57cec921e1b22be77255ac888c03b32"}
Nov 03 11:30:17.746741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.746711    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"]
Nov 03 11:30:17.913943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.902852    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:17.951639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.947236    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799b4527-5c0d-40c7-9e8b-03417efc5d9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b4527-5c0d-40c7-9e8b-03417efc5d9c-maas-api-service-ca\") pod \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\" (UID: \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\") "
Nov 03 11:30:17.951639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.947415    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799b4527-5c0d-40c7-9e8b-03417efc5d9c-kube-api-access-6s4gb\" (UniqueName: \"kubernetes.io/projected/799b4527-5c0d-40c7-9e8b-03417efc5d9c-kube-api-access-6s4gb\") pod \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\" (UID: \"799b4527-5c0d-40c7-9e8b-03417efc5d9c\") "
Nov 03 11:30:17.951891 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.951702    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799b4527-5c0d-40c7-9e8b-03417efc5d9c-kube-api-access-6s4gb" pod "799b4527-5c0d-40c7-9e8b-03417efc5d9c" (UID: "799b4527-5c0d-40c7-9e8b-03417efc5d9c"). InnerVolumeSpecName "kube-api-access-6s4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:17.952193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:17.952169    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799b4527-5c0d-40c7-9e8b-03417efc5d9c-maas-api-service-ca" pod "799b4527-5c0d-40c7-9e8b-03417efc5d9c" (UID: "799b4527-5c0d-40c7-9e8b-03417efc5d9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:18.048944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.048841    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s4gb\" (UniqueName: \"kubernetes.io/projected/799b4527-5c0d-40c7-9e8b-03417efc5d9c-kube-api-access-6s4gb\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.048944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.048869    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/799b4527-5c0d-40c7-9e8b-03417efc5d9c-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.223052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.223020    5393 scope.go:122] "RemoveContainer" containerID="d008d4d5d16d14ac987bd25f23e59fdc3280c7f7279117d846354be921982698"
Nov 03 11:30:18.223222 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.223156    5393 scope.go:122] "RemoveContainer" containerID="d07bc8049ad1731e24ff663f6e45684b97f77f37eaee73b4f32360e2e26d59d3"
Nov 03 11:30:18.232715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.232683    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babd4cd3-b063-475c-adc0-792eff654a76" path="/var/lib/kubelet/pods/babd4cd3-b063-475c-adc0-792eff654a76/volumes"
Nov 03 11:30:18.749364 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.749167    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" event={"ID":"9471f345-99b5-4ca1-b2fb-1995db880fda","Type":"ContainerStarted","Data":"0a3f7c6e449d450582b668dc0f8310dcd879973f7ced48a7508af7dd0e0236c2"}
Nov 03 11:30:18.752229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.752198    5393 generic.go:356] "Generic (PLEG): container finished" podID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerID="caae155eac956535c6877d04fdecb2446f7cb59e8aa267c60241900df651596c" exitCode=60
Nov 03 11:30:18.752390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.752257    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" event={"ID":"7171db8f-769b-4dba-9e97-31b1a5ed949d","Type":"ContainerDied","Data":"caae155eac956535c6877d04fdecb2446f7cb59e8aa267c60241900df651596c"}
Nov 03 11:30:18.752390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.752333    5393 scope.go:122] "RemoveContainer" containerID="db9cae4efe218c8fa92192b6e2b055ff6d82710757258140d8324764ab9d8d11"
Nov 03 11:30:18.752822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.752792    5393 scope.go:122] "RemoveContainer" containerID="caae155eac956535c6877d04fdecb2446f7cb59e8aa267c60241900df651596c"
Nov 03 11:30:18.753312 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:18.753283    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps_odh-ai-gateway-infra(7171db8f-769b-4dba-9e97-31b1a5ed949d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d"
Nov 03 11:30:18.757966 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.757937    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" event={"ID":"049a4cb6-866c-4d73-a260-7383b3fa0891","Type":"ContainerStarted","Data":"77cca2a3a0dc0ea0d154c30548c8e996a1b315d1830c9828a7ecb02c1af6adf9"}
Nov 03 11:30:18.760035 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.760005    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6" event={"ID":"799b4527-5c0d-40c7-9e8b-03417efc5d9c","Type":"ContainerDied","Data":"d4680ea2b12092dbb4c9e283268cae698ff3982bf547733900bbef8cd6417d44"}
Nov 03 11:30:18.760150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.760126    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"
Nov 03 11:30:18.764912 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.764885    5393 generic.go:356] "Generic (PLEG): container finished" podID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerID="16d0845f74f84c5925bf6753809b1566f57cec921e1b22be77255ac888c03b32" exitCode=60
Nov 03 11:30:18.765115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.765094    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" event={"ID":"78ff459d-34b6-440e-afe0-061e1d7860a9","Type":"ContainerDied","Data":"16d0845f74f84c5925bf6753809b1566f57cec921e1b22be77255ac888c03b32"}
Nov 03 11:30:18.765508 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.765421    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"]
Nov 03 11:30:18.765508 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.765451    5393 scope.go:122] "RemoveContainer" containerID="16d0845f74f84c5925bf6753809b1566f57cec921e1b22be77255ac888c03b32"
Nov 03 11:30:18.767155 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:18.765767    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt_odh-ai-gateway-infra(78ff459d-34b6-440e-afe0-061e1d7860a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9"
Nov 03 11:30:18.782979 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.781863    5393 scope.go:122] "RemoveContainer" containerID="799478c24934fdb772523c0be12a5e08d1aa5909c6796a374f1bc8d117096e40"
Nov 03 11:30:18.788442 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.788415    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"]
Nov 03 11:30:18.806126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.806101    5393 scope.go:122] "RemoveContainer" containerID="6334c5e82f920c22420071540abb7755f7a161347d62b19516954c8493941c51"
Nov 03 11:30:18.893065 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.893017    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"]
Nov 03 11:30:18.896300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.896272    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895090-5rts6"]
Nov 03 11:30:18.937434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.937409    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:18.965184 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.965148    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639d6260-0fdd-4698-83bc-61060581644f-kube-api-access-kjdr5\" (UniqueName: \"kubernetes.io/projected/639d6260-0fdd-4698-83bc-61060581644f-kube-api-access-kjdr5\") pod \"639d6260-0fdd-4698-83bc-61060581644f\" (UID: \"639d6260-0fdd-4698-83bc-61060581644f\") "
Nov 03 11:30:18.965386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.965202    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639d6260-0fdd-4698-83bc-61060581644f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639d6260-0fdd-4698-83bc-61060581644f-maas-api-service-ca\") pod \"639d6260-0fdd-4698-83bc-61060581644f\" (UID: \"639d6260-0fdd-4698-83bc-61060581644f\") "
Nov 03 11:30:18.965609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.965585    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639d6260-0fdd-4698-83bc-61060581644f-maas-api-service-ca" pod "639d6260-0fdd-4698-83bc-61060581644f" (UID: "639d6260-0fdd-4698-83bc-61060581644f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:18.965989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.965963    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/639d6260-0fdd-4698-83bc-61060581644f-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:18.967823 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:18.967801    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639d6260-0fdd-4698-83bc-61060581644f-kube-api-access-kjdr5" pod "639d6260-0fdd-4698-83bc-61060581644f" (UID: "639d6260-0fdd-4698-83bc-61060581644f"). InnerVolumeSpecName "kube-api-access-kjdr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:19.067643 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.067562    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjdr5\" (UniqueName: \"kubernetes.io/projected/639d6260-0fdd-4698-83bc-61060581644f-kube-api-access-kjdr5\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:19.082853 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.082782    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966" podUID="279ea1d2-0719-4d7f-bcf8-99f9d8f19270" containerName="main" containerID="cri-o://53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4" gracePeriod=45
Nov 03 11:30:19.223154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.223127    5393 scope.go:122] "RemoveContainer" containerID="f7fe0fe8b781f0c7bd9b1b12829c262ec840b0a8599711b51ffa7a6de6e97224"
Nov 03 11:30:19.369408 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.369377    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:30:19.474272 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474238    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-home\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-home\") pod \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") "
Nov 03 11:30:19.474473 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474315    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-model-cache\") pod \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") "
Nov 03 11:30:19.474473 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474362    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tmp-dir\") pod \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") "
Nov 03 11:30:19.474612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474475    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-dshm\") pod \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") "
Nov 03 11:30:19.474612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474547    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-kube-api-access-q4694\" (UniqueName: \"kubernetes.io/projected/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-kube-api-access-q4694\") pod \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") "
Nov 03 11:30:19.474612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474552    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-home" pod "279ea1d2-0719-4d7f-bcf8-99f9d8f19270" (UID: "279ea1d2-0719-4d7f-bcf8-99f9d8f19270"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:19.474765 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474607    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-model-cache" pod "279ea1d2-0719-4d7f-bcf8-99f9d8f19270" (UID: "279ea1d2-0719-4d7f-bcf8-99f9d8f19270"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:19.474765 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474641    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tmp-dir" pod "279ea1d2-0719-4d7f-bcf8-99f9d8f19270" (UID: "279ea1d2-0719-4d7f-bcf8-99f9d8f19270"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:19.474765 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.474665    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tls-certs\" (UniqueName: \"kubernetes.io/secret/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tls-certs\") pod \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\" (UID: \"279ea1d2-0719-4d7f-bcf8-99f9d8f19270\") "
Nov 03 11:30:19.475163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.475141    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:19.475428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.475167    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:19.475428 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.475180    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:19.477044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.476990    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-kube-api-access-q4694" pod "279ea1d2-0719-4d7f-bcf8-99f9d8f19270" (UID: "279ea1d2-0719-4d7f-bcf8-99f9d8f19270"). InnerVolumeSpecName "kube-api-access-q4694". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:19.477149 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.477090    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tls-certs" pod "279ea1d2-0719-4d7f-bcf8-99f9d8f19270" (UID: "279ea1d2-0719-4d7f-bcf8-99f9d8f19270"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:19.477149 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.477111    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-dshm" pod "279ea1d2-0719-4d7f-bcf8-99f9d8f19270" (UID: "279ea1d2-0719-4d7f-bcf8-99f9d8f19270"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:19.576437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.576401    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:19.576437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.576440    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4694\" (UniqueName: \"kubernetes.io/projected/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-kube-api-access-q4694\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:19.576722 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.576456    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279ea1d2-0719-4d7f-bcf8-99f9d8f19270-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:19.780626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.780591    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"]
Nov 03 11:30:19.783822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.783785    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" event={"ID":"e7e50e5e-7b9a-4a24-9def-949fed6aca68","Type":"ContainerStarted","Data":"d4c8e94b4bb2ff494851f56b7b21fc4a6687c6bb38d1ea3091bf9a4c844f560d"}
Nov 03 11:30:19.788630 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.788596    5393 generic.go:356] "Generic (PLEG): container finished" podID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerID="0a3f7c6e449d450582b668dc0f8310dcd879973f7ced48a7508af7dd0e0236c2" exitCode=60
Nov 03 11:30:19.788767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.788701    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" event={"ID":"9471f345-99b5-4ca1-b2fb-1995db880fda","Type":"ContainerDied","Data":"0a3f7c6e449d450582b668dc0f8310dcd879973f7ced48a7508af7dd0e0236c2"}
Nov 03 11:30:19.788767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.788763    5393 scope.go:122] "RemoveContainer" containerID="d07bc8049ad1731e24ff663f6e45684b97f77f37eaee73b4f32360e2e26d59d3"
Nov 03 11:30:19.792809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.792776    5393 generic.go:356] "Generic (PLEG): container finished" podID="279ea1d2-0719-4d7f-bcf8-99f9d8f19270" containerID="53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4" exitCode=0
Nov 03 11:30:19.792932 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.792890    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966" event={"ID":"279ea1d2-0719-4d7f-bcf8-99f9d8f19270","Type":"ContainerDied","Data":"53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4"}
Nov 03 11:30:19.793003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.792949    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966" event={"ID":"279ea1d2-0719-4d7f-bcf8-99f9d8f19270","Type":"ContainerDied","Data":"574cfc4b921bbd578726260dad8008c8258f7191557e658ea21ac798fb1964c2"}
Nov 03 11:30:19.793064 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.793042    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"
Nov 03 11:30:19.809506 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.809467    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq" event={"ID":"639d6260-0fdd-4698-83bc-61060581644f","Type":"ContainerDied","Data":"af7044996f80f17aa13e4f5828bd4ac9bf8200bd967e8955caebfd81e7b2c973"}
Nov 03 11:30:19.809714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.809676    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"
Nov 03 11:30:19.815577 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.815537    5393 generic.go:356] "Generic (PLEG): container finished" podID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerID="77cca2a3a0dc0ea0d154c30548c8e996a1b315d1830c9828a7ecb02c1af6adf9" exitCode=60
Nov 03 11:30:19.815709 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.815689    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" event={"ID":"049a4cb6-866c-4d73-a260-7383b3fa0891","Type":"ContainerDied","Data":"77cca2a3a0dc0ea0d154c30548c8e996a1b315d1830c9828a7ecb02c1af6adf9"}
Nov 03 11:30:19.816185 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.816156    5393 scope.go:122] "RemoveContainer" containerID="77cca2a3a0dc0ea0d154c30548c8e996a1b315d1830c9828a7ecb02c1af6adf9"
Nov 03 11:30:19.816358 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.816303    5393 scope.go:122] "RemoveContainer" containerID="53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4"
Nov 03 11:30:19.829219 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:19.829179    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k_odh-ai-gateway-infra(049a4cb6-866c-4d73-a260-7383b3fa0891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891"
Nov 03 11:30:19.835794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.835774    5393 scope.go:122] "RemoveContainer" containerID="53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4"
Nov 03 11:30:19.836438 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:19.836414    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4\": container with ID starting with 53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4 not found: ID does not exist" containerID="53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4"
Nov 03 11:30:19.836564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.836450    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4"} err="failed to get container status \"53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4\": rpc error: code = NotFound desc = could not find container \"53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4\": container with ID starting with 53332894476a76d300a8c728dcf41dd5f2ea3f152f5b092450bd04b9cfccf7a4 not found: ID does not exist"
Nov 03 11:30:19.836564 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.836471    5393 scope.go:122] "RemoveContainer" containerID="c0c8a7a0a46fe1ef72166a0e9f16948e7085e0131ccf712fcbd977bf15f333e9"
Nov 03 11:30:19.856018 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.855790    5393 scope.go:122] "RemoveContainer" containerID="d008d4d5d16d14ac987bd25f23e59fdc3280c7f7279117d846354be921982698"
Nov 03 11:30:19.905864 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.902873    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"]
Nov 03 11:30:19.913703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.910130    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56687c-29895090-dpxmq"]
Nov 03 11:30:19.917973 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.914954    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"]
Nov 03 11:30:19.937037 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.937006    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"]
Nov 03 11:30:19.938994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.938861    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-9c53b2/sub-test-model-9c53b2-kserve-5f56f9fc8f-d5966"]
Nov 03 11:30:19.981652 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:19.981622    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:20.027649 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.027619    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:20.074246 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.074215    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:20.103204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.103174    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ff459d-34b6-440e-afe0-061e1d7860a9-kube-api-access-p8kht\" (UniqueName: \"kubernetes.io/projected/78ff459d-34b6-440e-afe0-061e1d7860a9-kube-api-access-p8kht\") pod \"78ff459d-34b6-440e-afe0-061e1d7860a9\" (UID: \"78ff459d-34b6-440e-afe0-061e1d7860a9\") "
Nov 03 11:30:20.103403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.103239    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9471f345-99b5-4ca1-b2fb-1995db880fda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471f345-99b5-4ca1-b2fb-1995db880fda-maas-api-service-ca\") pod \"9471f345-99b5-4ca1-b2fb-1995db880fda\" (UID: \"9471f345-99b5-4ca1-b2fb-1995db880fda\") "
Nov 03 11:30:20.103403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.103300    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ff459d-34b6-440e-afe0-061e1d7860a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ff459d-34b6-440e-afe0-061e1d7860a9-maas-api-service-ca\") pod \"78ff459d-34b6-440e-afe0-061e1d7860a9\" (UID: \"78ff459d-34b6-440e-afe0-061e1d7860a9\") "
Nov 03 11:30:20.103403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.103363    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9471f345-99b5-4ca1-b2fb-1995db880fda-kube-api-access-rxm2j\" (UniqueName: \"kubernetes.io/projected/9471f345-99b5-4ca1-b2fb-1995db880fda-kube-api-access-rxm2j\") pod \"9471f345-99b5-4ca1-b2fb-1995db880fda\" (UID: \"9471f345-99b5-4ca1-b2fb-1995db880fda\") "
Nov 03 11:30:20.104013 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.103986    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9471f345-99b5-4ca1-b2fb-1995db880fda-maas-api-service-ca" pod "9471f345-99b5-4ca1-b2fb-1995db880fda" (UID: "9471f345-99b5-4ca1-b2fb-1995db880fda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:20.104321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.104297    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ff459d-34b6-440e-afe0-061e1d7860a9-maas-api-service-ca" pod "78ff459d-34b6-440e-afe0-061e1d7860a9" (UID: "78ff459d-34b6-440e-afe0-061e1d7860a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:20.106279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.106256    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9471f345-99b5-4ca1-b2fb-1995db880fda-kube-api-access-rxm2j" pod "9471f345-99b5-4ca1-b2fb-1995db880fda" (UID: "9471f345-99b5-4ca1-b2fb-1995db880fda"). InnerVolumeSpecName "kube-api-access-rxm2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:20.107085 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.107058    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ff459d-34b6-440e-afe0-061e1d7860a9-kube-api-access-p8kht" pod "78ff459d-34b6-440e-afe0-061e1d7860a9" (UID: "78ff459d-34b6-440e-afe0-061e1d7860a9"). InnerVolumeSpecName "kube-api-access-p8kht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:20.204699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.204669    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7171db8f-769b-4dba-9e97-31b1a5ed949d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7171db8f-769b-4dba-9e97-31b1a5ed949d-maas-api-service-ca\") pod \"7171db8f-769b-4dba-9e97-31b1a5ed949d\" (UID: \"7171db8f-769b-4dba-9e97-31b1a5ed949d\") "
Nov 03 11:30:20.204870 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.204784    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7171db8f-769b-4dba-9e97-31b1a5ed949d-kube-api-access-jcddg\" (UniqueName: \"kubernetes.io/projected/7171db8f-769b-4dba-9e97-31b1a5ed949d-kube-api-access-jcddg\") pod \"7171db8f-769b-4dba-9e97-31b1a5ed949d\" (UID: \"7171db8f-769b-4dba-9e97-31b1a5ed949d\") "
Nov 03 11:30:20.205215 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.205182    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7171db8f-769b-4dba-9e97-31b1a5ed949d-maas-api-service-ca" pod "7171db8f-769b-4dba-9e97-31b1a5ed949d" (UID: "7171db8f-769b-4dba-9e97-31b1a5ed949d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:20.205338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.205321    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8kht\" (UniqueName: \"kubernetes.io/projected/78ff459d-34b6-440e-afe0-061e1d7860a9-kube-api-access-p8kht\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.205382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.205345    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471f345-99b5-4ca1-b2fb-1995db880fda-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.205382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.205360    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ff459d-34b6-440e-afe0-061e1d7860a9-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.205382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.205374    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxm2j\" (UniqueName: \"kubernetes.io/projected/9471f345-99b5-4ca1-b2fb-1995db880fda-kube-api-access-rxm2j\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.205471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.205387    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7171db8f-769b-4dba-9e97-31b1a5ed949d-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.206996 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.206973    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7171db8f-769b-4dba-9e97-31b1a5ed949d-kube-api-access-jcddg" pod "7171db8f-769b-4dba-9e97-31b1a5ed949d" (UID: "7171db8f-769b-4dba-9e97-31b1a5ed949d"). InnerVolumeSpecName "kube-api-access-jcddg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:20.231616 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.231572    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279ea1d2-0719-4d7f-bcf8-99f9d8f19270" path="/var/lib/kubelet/pods/279ea1d2-0719-4d7f-bcf8-99f9d8f19270/volumes"
Nov 03 11:30:20.232153 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.232130    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639d6260-0fdd-4698-83bc-61060581644f" path="/var/lib/kubelet/pods/639d6260-0fdd-4698-83bc-61060581644f/volumes"
Nov 03 11:30:20.232595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.232576    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" path="/var/lib/kubelet/pods/799b4527-5c0d-40c7-9e8b-03417efc5d9c/volumes"
Nov 03 11:30:20.233375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.233348    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq" podUID="a07ec5c1-d744-43cc-96f1-de34b47a99fd" containerName="main" containerID="cri-o://936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7" gracePeriod=45
Nov 03 11:30:20.306825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.306783    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcddg\" (UniqueName: \"kubernetes.io/projected/7171db8f-769b-4dba-9e97-31b1a5ed949d-kube-api-access-jcddg\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.551207 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.551178    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:30:20.610696 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.610659    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-home\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-home\") pod \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") "
Nov 03 11:30:20.610878 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.610728    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-model-cache\") pod \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") "
Nov 03 11:30:20.610999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.610882    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07ec5c1-d744-43cc-96f1-de34b47a99fd-kube-api-access-bgnvx\" (UniqueName: \"kubernetes.io/projected/a07ec5c1-d744-43cc-96f1-de34b47a99fd-kube-api-access-bgnvx\") pod \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") "
Nov 03 11:30:20.610999 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.610910    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tmp-dir\") pod \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") "
Nov 03 11:30:20.611157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.611019    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-model-cache" pod "a07ec5c1-d744-43cc-96f1-de34b47a99fd" (UID: "a07ec5c1-d744-43cc-96f1-de34b47a99fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:20.611157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.611037    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-home" pod "a07ec5c1-d744-43cc-96f1-de34b47a99fd" (UID: "a07ec5c1-d744-43cc-96f1-de34b47a99fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:20.611157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.611049    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-dshm\") pod \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") "
Nov 03 11:30:20.611157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.611084    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tls-certs\") pod \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\" (UID: \"a07ec5c1-d744-43cc-96f1-de34b47a99fd\") "
Nov 03 11:30:20.611441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.611257    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tmp-dir" pod "a07ec5c1-d744-43cc-96f1-de34b47a99fd" (UID: "a07ec5c1-d744-43cc-96f1-de34b47a99fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:20.611880 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.611744    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.612014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.611884    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.612014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.611900    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.613942 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.613898    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-dshm" pod "a07ec5c1-d744-43cc-96f1-de34b47a99fd" (UID: "a07ec5c1-d744-43cc-96f1-de34b47a99fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:30:20.614061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.613966    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tls-certs" pod "a07ec5c1-d744-43cc-96f1-de34b47a99fd" (UID: "a07ec5c1-d744-43cc-96f1-de34b47a99fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:20.614061 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.613992    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07ec5c1-d744-43cc-96f1-de34b47a99fd-kube-api-access-bgnvx" pod "a07ec5c1-d744-43cc-96f1-de34b47a99fd" (UID: "a07ec5c1-d744-43cc-96f1-de34b47a99fd"). InnerVolumeSpecName "kube-api-access-bgnvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:20.712896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.712856    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnvx\" (UniqueName: \"kubernetes.io/projected/a07ec5c1-d744-43cc-96f1-de34b47a99fd-kube-api-access-bgnvx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.712896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.712896    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07ec5c1-d744-43cc-96f1-de34b47a99fd-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.713165 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.712911    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07ec5c1-d744-43cc-96f1-de34b47a99fd-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:20.727029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.726992    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns" podUID="1441f96d-52b2-420c-bb88-988f13c65e65" containerName="payload-pre-processing" containerID="cri-o://fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d" gracePeriod=25
Nov 03 11:30:20.828681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.828602    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"
Nov 03 11:30:20.829172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.828606    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj" event={"ID":"9471f345-99b5-4ca1-b2fb-1995db880fda","Type":"ContainerDied","Data":"2b073b9d1cf9dfaf5c695829d4b9a2c149bd86bf3ed006a8a5c7a823a40f103c"}
Nov 03 11:30:20.829172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.828754    5393 scope.go:122] "RemoveContainer" containerID="0a3f7c6e449d450582b668dc0f8310dcd879973f7ced48a7508af7dd0e0236c2"
Nov 03 11:30:20.831448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.831417    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps" event={"ID":"7171db8f-769b-4dba-9e97-31b1a5ed949d","Type":"ContainerDied","Data":"08ed580cf4df5ea2f1de3d6098bb69de6387a6383df12fb38f3b185a8cdde968"}
Nov 03 11:30:20.831605 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.831470    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"
Nov 03 11:30:20.839211 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.839186    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt" event={"ID":"78ff459d-34b6-440e-afe0-061e1d7860a9","Type":"ContainerDied","Data":"1ac3814c17946f24e206f0a8e8c7b7008a2e0664cb5b26af9a35d13eb97df042"}
Nov 03 11:30:20.839337 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.839196    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"
Nov 03 11:30:20.840293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.840271    5393 scope.go:122] "RemoveContainer" containerID="caae155eac956535c6877d04fdecb2446f7cb59e8aa267c60241900df651596c"
Nov 03 11:30:20.843556 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.843515    5393 generic.go:356] "Generic (PLEG): container finished" podID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerID="d4c8e94b4bb2ff494851f56b7b21fc4a6687c6bb38d1ea3091bf9a4c844f560d" exitCode=60
Nov 03 11:30:20.843672 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.843555    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" event={"ID":"e7e50e5e-7b9a-4a24-9def-949fed6aca68","Type":"ContainerDied","Data":"d4c8e94b4bb2ff494851f56b7b21fc4a6687c6bb38d1ea3091bf9a4c844f560d"}
Nov 03 11:30:20.844162 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.844129    5393 scope.go:122] "RemoveContainer" containerID="d4c8e94b4bb2ff494851f56b7b21fc4a6687c6bb38d1ea3091bf9a4c844f560d"
Nov 03 11:30:20.844441 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:20.844413    5393 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4_odh-ai-gateway-infra(e7e50e5e-7b9a-4a24-9def-949fed6aca68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68"
Nov 03 11:30:20.845206 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.845179    5393 generic.go:356] "Generic (PLEG): container finished" podID="a07ec5c1-d744-43cc-96f1-de34b47a99fd" containerID="936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7" exitCode=0
Nov 03 11:30:20.845318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.845216    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"
Nov 03 11:30:20.845318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.845260    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq" event={"ID":"a07ec5c1-d744-43cc-96f1-de34b47a99fd","Type":"ContainerDied","Data":"936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7"}
Nov 03 11:30:20.845318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.845296    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq" event={"ID":"a07ec5c1-d744-43cc-96f1-de34b47a99fd","Type":"ContainerDied","Data":"5f17d699b106f5f3b80b2ba6de9a8b13139958c2c0dd2aec9bc8cdc50a50eae7"}
Nov 03 11:30:20.853995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.853958    5393 scope.go:122] "RemoveContainer" containerID="16d0845f74f84c5925bf6753809b1566f57cec921e1b22be77255ac888c03b32"
Nov 03 11:30:20.861780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.861754    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"]
Nov 03 11:30:20.868503 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.868477    5393 scope.go:122] "RemoveContainer" containerID="f7fe0fe8b781f0c7bd9b1b12829c262ec840b0a8599711b51ffa7a6de6e97224"
Nov 03 11:30:20.871269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.871245    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"]
Nov 03 11:30:20.875388 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.875357    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cc8214-29895090-6njps"]
Nov 03 11:30:20.890114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.890087    5393 scope.go:122] "RemoveContainer" containerID="936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7"
Nov 03 11:30:20.892804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.892779    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"]
Nov 03 11:30:20.899042 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.899021    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010232-29895090-ks2pj"]
Nov 03 11:30:20.905248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.905235    5393 scope.go:122] "RemoveContainer" containerID="936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7"
Nov 03 11:30:20.905666 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:20.905639    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7\": container with ID starting with 936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7 not found: ID does not exist" containerID="936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7"
Nov 03 11:30:20.905745 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.905705    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7"} err="failed to get container status \"936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7\": rpc error: code = NotFound desc = could not find container \"936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7\": container with ID starting with 936e28309be798f6072c165f1faa1b0da9eeda0c2fe044d93fafa8b095b861d7 not found: ID does not exist"
Nov 03 11:30:20.906501 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.906478    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"]
Nov 03 11:30:20.910623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.910443    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f78621-29895090-rkljt"]
Nov 03 11:30:20.922874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.922854    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"]
Nov 03 11:30:20.926802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.926779    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-ef1662/sub-test-model-ef1662-kserve-5bb4477b49-cqvhq"]
Nov 03 11:30:20.988440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:20.988413    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:21.125706 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.125674    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049a4cb6-866c-4d73-a260-7383b3fa0891-kube-api-access-vbjfg\" (UniqueName: \"kubernetes.io/projected/049a4cb6-866c-4d73-a260-7383b3fa0891-kube-api-access-vbjfg\") pod \"049a4cb6-866c-4d73-a260-7383b3fa0891\" (UID: \"049a4cb6-866c-4d73-a260-7383b3fa0891\") "
Nov 03 11:30:21.125879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.125758    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049a4cb6-866c-4d73-a260-7383b3fa0891-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049a4cb6-866c-4d73-a260-7383b3fa0891-maas-api-service-ca\") pod \"049a4cb6-866c-4d73-a260-7383b3fa0891\" (UID: \"049a4cb6-866c-4d73-a260-7383b3fa0891\") "
Nov 03 11:30:21.126177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.126147    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049a4cb6-866c-4d73-a260-7383b3fa0891-maas-api-service-ca" pod "049a4cb6-866c-4d73-a260-7383b3fa0891" (UID: "049a4cb6-866c-4d73-a260-7383b3fa0891"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:21.126307 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.126293    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/049a4cb6-866c-4d73-a260-7383b3fa0891-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:21.128115 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.128095    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049a4cb6-866c-4d73-a260-7383b3fa0891-kube-api-access-vbjfg" pod "049a4cb6-866c-4d73-a260-7383b3fa0891" (UID: "049a4cb6-866c-4d73-a260-7383b3fa0891"). InnerVolumeSpecName "kube-api-access-vbjfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:21.227189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.227156    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbjfg\" (UniqueName: \"kubernetes.io/projected/049a4cb6-866c-4d73-a260-7383b3fa0891-kube-api-access-vbjfg\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:21.855235 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.855206    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"
Nov 03 11:30:21.855720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.855351    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k" event={"ID":"049a4cb6-866c-4d73-a260-7383b3fa0891","Type":"ContainerDied","Data":"6b90c49f6c8956193b750d0cb1c2539a301b28a034b6763e3320bf0324618bf0"}
Nov 03 11:30:21.855720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.855487    5393 scope.go:122] "RemoveContainer" containerID="77cca2a3a0dc0ea0d154c30548c8e996a1b315d1830c9828a7ecb02c1af6adf9"
Nov 03 11:30:21.906494 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.906460    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"]
Nov 03 11:30:21.908248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:21.908225    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-917da2-29895090-8mx8k"]
Nov 03 11:30:22.025758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.025722    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:22.143044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.142952    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e50e5e-7b9a-4a24-9def-949fed6aca68-kube-api-access-nvln7\" (UniqueName: \"kubernetes.io/projected/e7e50e5e-7b9a-4a24-9def-949fed6aca68-kube-api-access-nvln7\") pod \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\" (UID: \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\") "
Nov 03 11:30:22.143197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.143183    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e50e5e-7b9a-4a24-9def-949fed6aca68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e50e5e-7b9a-4a24-9def-949fed6aca68-maas-api-service-ca\") pod \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\" (UID: \"e7e50e5e-7b9a-4a24-9def-949fed6aca68\") "
Nov 03 11:30:22.143521 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.143498    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e50e5e-7b9a-4a24-9def-949fed6aca68-maas-api-service-ca" pod "e7e50e5e-7b9a-4a24-9def-949fed6aca68" (UID: "e7e50e5e-7b9a-4a24-9def-949fed6aca68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:22.143624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.143607    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e50e5e-7b9a-4a24-9def-949fed6aca68-maas-api-service-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:22.145522 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.145492    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e50e5e-7b9a-4a24-9def-949fed6aca68-kube-api-access-nvln7" pod "e7e50e5e-7b9a-4a24-9def-949fed6aca68" (UID: "e7e50e5e-7b9a-4a24-9def-949fed6aca68"). InnerVolumeSpecName "kube-api-access-nvln7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:22.232182 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.232147    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" path="/var/lib/kubelet/pods/049a4cb6-866c-4d73-a260-7383b3fa0891/volumes"
Nov 03 11:30:22.232840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.232817    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" path="/var/lib/kubelet/pods/7171db8f-769b-4dba-9e97-31b1a5ed949d/volumes"
Nov 03 11:30:22.233291 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.233272    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" path="/var/lib/kubelet/pods/78ff459d-34b6-440e-afe0-061e1d7860a9/volumes"
Nov 03 11:30:22.234150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.234129    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" path="/var/lib/kubelet/pods/9471f345-99b5-4ca1-b2fb-1995db880fda/volumes"
Nov 03 11:30:22.234599 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.234581    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07ec5c1-d744-43cc-96f1-de34b47a99fd" path="/var/lib/kubelet/pods/a07ec5c1-d744-43cc-96f1-de34b47a99fd/volumes"
Nov 03 11:30:22.244969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.244950    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvln7\" (UniqueName: \"kubernetes.io/projected/e7e50e5e-7b9a-4a24-9def-949fed6aca68-kube-api-access-nvln7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:22.879310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.879271    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4" event={"ID":"e7e50e5e-7b9a-4a24-9def-949fed6aca68","Type":"ContainerDied","Data":"9db14ae26c9a218d974a90789326ea99cdb8297fb5f15c19ddf797e99b98c38d"}
Nov 03 11:30:22.879772 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.879343    5393 scope.go:122] "RemoveContainer" containerID="d4c8e94b4bb2ff494851f56b7b21fc4a6687c6bb38d1ea3091bf9a4c844f560d"
Nov 03 11:30:22.879772 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.879452    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"
Nov 03 11:30:22.907123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.905173    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"]
Nov 03 11:30:22.909828 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:22.907358    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b1c6f6-29895090-m5nj4"]
Nov 03 11:30:24.231766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:24.231726    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" path="/var/lib/kubelet/pods/e7e50e5e-7b9a-4a24-9def-949fed6aca68/volumes"
Nov 03 11:30:26.878461 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:26.878418    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns" podUID="1441f96d-52b2-420c-bb88-988f13c65e65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:30:36.878158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:36.878120    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns" podUID="1441f96d-52b2-420c-bb88-988f13c65e65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:30:41.011221 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.011193    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:30:41.086127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.086088    5393 generic.go:356] "Generic (PLEG): container finished" podID="1441f96d-52b2-420c-bb88-988f13c65e65" containerID="fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d" exitCode=0
Nov 03 11:30:41.086310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.086197    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"
Nov 03 11:30:41.086310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.086205    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns" event={"ID":"1441f96d-52b2-420c-bb88-988f13c65e65","Type":"ContainerDied","Data":"fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d"}
Nov 03 11:30:41.086310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.086264    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns" event={"ID":"1441f96d-52b2-420c-bb88-988f13c65e65","Type":"ContainerDied","Data":"396ad3359d0e9f19342296f7eda678125bc8b7e52c7f21d96eb253243eae0d5f"}
Nov 03 11:30:41.086310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.086288    5393 scope.go:122] "RemoveContainer" containerID="fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d"
Nov 03 11:30:41.096789 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.096767    5393 scope.go:122] "RemoveContainer" containerID="fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d"
Nov 03 11:30:41.097084 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:41.097065    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d\": container with ID starting with fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d not found: ID does not exist" containerID="fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d"
Nov 03 11:30:41.097147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.097092    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d"} err="failed to get container status \"fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d\": rpc error: code = NotFound desc = could not find container \"fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d\": container with ID starting with fe1bc4aa465214ed15f794de9334f2deb889d4551d282d132eed683a9d61b32d not found: ID does not exist"
Nov 03 11:30:41.146955 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.146905    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1441f96d-52b2-420c-bb88-988f13c65e65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1441f96d-52b2-420c-bb88-988f13c65e65-plugins-config-volume\") pod \"1441f96d-52b2-420c-bb88-988f13c65e65\" (UID: \"1441f96d-52b2-420c-bb88-988f13c65e65\") "
Nov 03 11:30:41.147139 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.147031    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1441f96d-52b2-420c-bb88-988f13c65e65-kube-api-access-chgkp\" (UniqueName: \"kubernetes.io/projected/1441f96d-52b2-420c-bb88-988f13c65e65-kube-api-access-chgkp\") pod \"1441f96d-52b2-420c-bb88-988f13c65e65\" (UID: \"1441f96d-52b2-420c-bb88-988f13c65e65\") "
Nov 03 11:30:41.147280 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.147259    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1441f96d-52b2-420c-bb88-988f13c65e65-plugins-config-volume" pod "1441f96d-52b2-420c-bb88-988f13c65e65" (UID: "1441f96d-52b2-420c-bb88-988f13c65e65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:41.147591 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.147571    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1441f96d-52b2-420c-bb88-988f13c65e65-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:41.149361 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.149335    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1441f96d-52b2-420c-bb88-988f13c65e65-kube-api-access-chgkp" pod "1441f96d-52b2-420c-bb88-988f13c65e65" (UID: "1441f96d-52b2-420c-bb88-988f13c65e65"). InnerVolumeSpecName "kube-api-access-chgkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:41.249751 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.249656    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chgkp\" (UniqueName: \"kubernetes.io/projected/1441f96d-52b2-420c-bb88-988f13c65e65-kube-api-access-chgkp\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:41.411688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.411653    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"]
Nov 03 11:30:41.415257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:41.415230    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-185908-78b5bdcbff-q72ns"]
Nov 03 11:30:42.231624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:42.231592    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1441f96d-52b2-420c-bb88-988f13c65e65" path="/var/lib/kubelet/pods/1441f96d-52b2-420c-bb88-988f13c65e65/volumes"
Nov 03 11:30:43.189430 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:30:43.189396    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:30:43.233058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:43.233019    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"]
Nov 03 11:30:43.279331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:43.278007    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"]
Nov 03 11:30:48.233812 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:48.233775    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws" podUID="f3a3de92-9886-41bd-99ab-2fa5ad62b841" containerName="payload-processing" containerID="cri-o://f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d" gracePeriod=25
Nov 03 11:30:48.278568 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:48.278529    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk" podUID="b087ec3a-d711-4e4f-b219-c6d026fa08c4" containerName="payload-pre-processing" containerID="cri-o://ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4" gracePeriod=25
Nov 03 11:30:51.180342 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:51.180301    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk" podUID="b087ec3a-d711-4e4f-b219-c6d026fa08c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:30:51.180342 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:51.180328    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws" podUID="f3a3de92-9886-41bd-99ab-2fa5ad62b841" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:30:55.510664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:55.510624    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"]
Nov 03 11:30:55.511192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:55.510879    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy" containerID="cri-o://5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506" gracePeriod=30
Nov 03 11:30:56.030078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030045    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"]
Nov 03 11:30:56.030619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030604    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030619    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030634    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030640    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030649    5393 cpu_manager.go:414] "containerMap: removing container" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030654    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030660    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030665    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030672    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1441f96d-52b2-420c-bb88-988f13c65e65" containerName="payload-pre-processing"
Nov 03 11:30:56.030678 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030676    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1441f96d-52b2-420c-bb88-988f13c65e65" containerName="payload-pre-processing"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030689    5393 cpu_manager.go:414] "containerMap: removing container" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030693    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030700    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030704    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030709    5393 cpu_manager.go:414] "containerMap: removing container" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030714    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030721    5393 cpu_manager.go:414] "containerMap: removing container" podUID="279ea1d2-0719-4d7f-bcf8-99f9d8f19270" containerName="main"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030725    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279ea1d2-0719-4d7f-bcf8-99f9d8f19270" containerName="main"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030732    5393 cpu_manager.go:414] "containerMap: removing container" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030736    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030743    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030747    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030753    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030757    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030768    5393 cpu_manager.go:414] "containerMap: removing container" podUID="babd4cd3-b063-475c-adc0-792eff654a76" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030772    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babd4cd3-b063-475c-adc0-792eff654a76" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030779    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030783    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030789    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030793    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030801    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b03af0f2-633e-48b4-ac32-f78749a60da3" containerName="revoke-keys"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030806    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03af0f2-633e-48b4-ac32-f78749a60da3" containerName="revoke-keys"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030812    5393 cpu_manager.go:414] "containerMap: removing container" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030816    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030833    5393 cpu_manager.go:414] "containerMap: removing container" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030837    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030844    5393 cpu_manager.go:414] "containerMap: removing container" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030848    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030856    5393 cpu_manager.go:414] "containerMap: removing container" podUID="babd4cd3-b063-475c-adc0-792eff654a76" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030860    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babd4cd3-b063-475c-adc0-792eff654a76" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030867    5393 cpu_manager.go:414] "containerMap: removing container" podUID="a07ec5c1-d744-43cc-96f1-de34b47a99fd" containerName="main"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030872    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07ec5c1-d744-43cc-96f1-de34b47a99fd" containerName="main"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030884    5393 cpu_manager.go:414] "containerMap: removing container" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030888    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030899    5393 cpu_manager.go:414] "containerMap: removing container" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030902    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.030997    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031006    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:30:56.030997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031013    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031019    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="babd4cd3-b063-475c-adc0-792eff654a76" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031026    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031031    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2977665-bb0a-4a29-a5a1-5bf515156271" containerName="main"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031037    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031042    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03af0f2-633e-48b4-ac32-f78749a60da3" containerName="revoke-keys"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031047    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031053    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031059    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031065    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="babd4cd3-b063-475c-adc0-792eff654a76" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031069    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07ec5c1-d744-43cc-96f1-de34b47a99fd" containerName="main"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031077    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031081    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031087    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031092    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031099    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031104    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031108    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1441f96d-52b2-420c-bb88-988f13c65e65" containerName="payload-pre-processing"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031113    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031118    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:30:56.032247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.031123    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="279ea1d2-0719-4d7f-bcf8-99f9d8f19270" containerName="main"
Nov 03 11:30:56.038157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.038125    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.042881 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.042850    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"]
Nov 03 11:30:56.059398 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.059368    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:30:56.059561 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.059425    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:30:56.118505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.118459    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lb2b\" (UniqueName: \"kubernetes.io/projected/d2a17c39-ad43-4a41-83c3-eca3245321f8-kube-api-access-8lb2b\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.118711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.118609    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.118711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.118676    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.220316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.220284    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lb2b\" (UniqueName: \"kubernetes.io/projected/d2a17c39-ad43-4a41-83c3-eca3245321f8-kube-api-access-8lb2b\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.220489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.220352    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.220489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.220400    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.221118 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.221094    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.223147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.223122    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.229949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.229899    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lb2b\" (UniqueName: \"kubernetes.io/projected/d2a17c39-ad43-4a41-83c3-eca3245321f8-kube-api-access-8lb2b\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.359134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.359097    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:56.638597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:56.638569    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"]
Nov 03 11:30:56.640373 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:30:56.640348    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a17c39_ad43_4a41_83c3_eca3245321f8.slice/crio-3e7ec8f0f3332afc1ec479d8d53d1e762501cf4e69319a44de5280b3331800c5 WatchSource:0}: Error finding container 3e7ec8f0f3332afc1ec479d8d53d1e762501cf4e69319a44de5280b3331800c5: Status 404 returned error can't find the container with id 3e7ec8f0f3332afc1ec479d8d53d1e762501cf4e69319a44de5280b3331800c5
Nov 03 11:30:57.210005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:57.209886    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x" event={"ID":"d2a17c39-ad43-4a41-83c3-eca3245321f8","Type":"ContainerStarted","Data":"bb92cae100c5628209561811963b8b21c4cb81d75067a5c7de8c497a29b882b5"}
Nov 03 11:30:57.210005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:57.209988    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x" event={"ID":"d2a17c39-ad43-4a41-83c3-eca3245321f8","Type":"ContainerStarted","Data":"3e7ec8f0f3332afc1ec479d8d53d1e762501cf4e69319a44de5280b3331800c5"}
Nov 03 11:30:57.228769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:57.228702    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x" podStartSLOduration=1.228682659 podStartE2EDuration="1.228682659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:30:57.227675443 +0000 UTC m=+3701.132529885" watchObservedRunningTime="2026-11-03 11:30:57.228682659 +0000 UTC m=+3701.133537102"
Nov 03 11:30:58.217529 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:58.217496    5393 generic.go:356] "Generic (PLEG): container finished" podID="d2a17c39-ad43-4a41-83c3-eca3245321f8" containerID="bb92cae100c5628209561811963b8b21c4cb81d75067a5c7de8c497a29b882b5" exitCode=0
Nov 03 11:30:58.218089 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:58.217582    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x" event={"ID":"d2a17c39-ad43-4a41-83c3-eca3245321f8","Type":"ContainerDied","Data":"bb92cae100c5628209561811963b8b21c4cb81d75067a5c7de8c497a29b882b5"}
Nov 03 11:30:59.385423 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.385396    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:30:59.481330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.481248    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-xks-ca\") pod \"d2a17c39-ad43-4a41-83c3-eca3245321f8\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") "
Nov 03 11:30:59.481330 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.481307    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a17c39-ad43-4a41-83c3-eca3245321f8-kube-api-access-8lb2b\" (UniqueName: \"kubernetes.io/projected/d2a17c39-ad43-4a41-83c3-eca3245321f8-kube-api-access-8lb2b\") pod \"d2a17c39-ad43-4a41-83c3-eca3245321f8\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") "
Nov 03 11:30:59.481534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.481474    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-openshift-ca\") pod \"d2a17c39-ad43-4a41-83c3-eca3245321f8\" (UID: \"d2a17c39-ad43-4a41-83c3-eca3245321f8\") "
Nov 03 11:30:59.481911 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.481877    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-openshift-ca" pod "d2a17c39-ad43-4a41-83c3-eca3245321f8" (UID: "d2a17c39-ad43-4a41-83c3-eca3245321f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:30:59.482084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.482041    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:59.483520 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.483487    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-xks-ca" pod "d2a17c39-ad43-4a41-83c3-eca3245321f8" (UID: "d2a17c39-ad43-4a41-83c3-eca3245321f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:30:59.483623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.483592    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a17c39-ad43-4a41-83c3-eca3245321f8-kube-api-access-8lb2b" pod "d2a17c39-ad43-4a41-83c3-eca3245321f8" (UID: "d2a17c39-ad43-4a41-83c3-eca3245321f8"). InnerVolumeSpecName "kube-api-access-8lb2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:30:59.583805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.583772    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a17c39-ad43-4a41-83c3-eca3245321f8-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:30:59.583805 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:30:59.583802    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lb2b\" (UniqueName: \"kubernetes.io/projected/d2a17c39-ad43-4a41-83c3-eca3245321f8-kube-api-access-8lb2b\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:00.233120 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.233089    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"
Nov 03 11:31:00.233322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.233125    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x" event={"ID":"d2a17c39-ad43-4a41-83c3-eca3245321f8","Type":"ContainerDied","Data":"3e7ec8f0f3332afc1ec479d8d53d1e762501cf4e69319a44de5280b3331800c5"}
Nov 03 11:31:00.233322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.233174    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e7ec8f0f3332afc1ec479d8d53d1e762501cf4e69319a44de5280b3331800c5"
Nov 03 11:31:00.789574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.789497    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:31:00.902134 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902102    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a9215b-e2bd-4992-a6d6-97add590eb3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9215b-e2bd-4992-a6d6-97add590eb3b-istiod-ca-cert\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902225    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-socket\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902246    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-envoy\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902287    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-podinfo\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902330    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-credential-socket\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902396    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-token\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-token\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902429    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-data\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902479    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-certs\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902566    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-socket" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:00.902657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902579    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-kube-api-access-gj2kx\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-kube-api-access-gj2kx\") pod \"41a9215b-e2bd-4992-a6d6-97add590eb3b\" (UID: \"41a9215b-e2bd-4992-a6d6-97add590eb3b\") "
Nov 03 11:31:00.902841 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902740    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-credential-socket" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:00.902841 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.902755    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-data" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:00.903089 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.903060    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-certs" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:00.903311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.903289    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:00.903434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.903380    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:00.903434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.903398    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:00.903434 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.903412    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:00.903626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.903513    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a9215b-e2bd-4992-a6d6-97add590eb3b-istiod-ca-cert" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:00.904819 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.904792    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-envoy" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:00.905003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.904981    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-token" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:00.905572 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.905541    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-kube-api-access-gj2kx" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "kube-api-access-gj2kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:00.905741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:00.905711    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-podinfo" pod "41a9215b-e2bd-4992-a6d6-97add590eb3b" (UID: "41a9215b-e2bd-4992-a6d6-97add590eb3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:31:01.004380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.004346    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9215b-e2bd-4992-a6d6-97add590eb3b-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:01.004380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.004377    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:01.004380 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.004388    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:01.004628 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.004396    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:01.004628 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.004406    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj2kx\" (UniqueName: \"kubernetes.io/projected/41a9215b-e2bd-4992-a6d6-97add590eb3b-kube-api-access-gj2kx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:01.179986 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.179913    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws" podUID="f3a3de92-9886-41bd-99ab-2fa5ad62b841" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:01.180202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.180023    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk" podUID="b087ec3a-d711-4e4f-b219-c6d026fa08c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:01.243366 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.243329    5393 generic.go:356] "Generic (PLEG): container finished" podID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerID="5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506" exitCode=0
Nov 03 11:31:01.243520 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.243386    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"
Nov 03 11:31:01.243520 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.243426    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" event={"ID":"41a9215b-e2bd-4992-a6d6-97add590eb3b","Type":"ContainerDied","Data":"5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506"}
Nov 03 11:31:01.243520 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.243500    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4" event={"ID":"41a9215b-e2bd-4992-a6d6-97add590eb3b","Type":"ContainerDied","Data":"9fef78212643b6aa62e419ec492d2942cd1ab075bc077223397e47c5db88bce1"}
Nov 03 11:31:01.243664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.243531    5393 scope.go:122] "RemoveContainer" containerID="5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506"
Nov 03 11:31:01.255199 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.255178    5393 scope.go:122] "RemoveContainer" containerID="74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e"
Nov 03 11:31:01.265419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.265396    5393 scope.go:122] "RemoveContainer" containerID="5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506"
Nov 03 11:31:01.265775 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:01.265742    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506\": container with ID starting with 5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506 not found: ID does not exist" containerID="5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506"
Nov 03 11:31:01.265869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.265784    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506"} err="failed to get container status \"5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506\": rpc error: code = NotFound desc = could not find container \"5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506\": container with ID starting with 5d20c64a28db7df866fd649e94d690c5009dcc80c6bfe50556fa8fd5c4b13506 not found: ID does not exist"
Nov 03 11:31:01.265869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.265801    5393 scope.go:122] "RemoveContainer" containerID="74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e"
Nov 03 11:31:01.266125 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:01.266108    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e\": container with ID starting with 74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e not found: ID does not exist" containerID="74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e"
Nov 03 11:31:01.266172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.266131    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e"} err="failed to get container status \"74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e\": rpc error: code = NotFound desc = could not find container \"74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e\": container with ID starting with 74d52c0a77956e1abde16f30b55ca7699ba4d3fcc964806dc844da69fbcbd31e not found: ID does not exist"
Nov 03 11:31:01.266479 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.266460    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"]
Nov 03 11:31:01.273740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:01.273707    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-56687c-openshift-default-5d54886fb9-w65m4"]
Nov 03 11:31:02.240533 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:02.240491    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" path="/var/lib/kubelet/pods/41a9215b-e2bd-4992-a6d6-97add590eb3b/volumes"
Nov 03 11:31:05.041119 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:05.041079    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"]
Nov 03 11:31:07.126832 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:07.126798    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"]
Nov 03 11:31:08.558130 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.558099    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:31:08.621951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.621899    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:31:08.636484 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.636455    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a3de92-9886-41bd-99ab-2fa5ad62b841-kube-api-access-6tfw9\" (UniqueName: \"kubernetes.io/projected/f3a3de92-9886-41bd-99ab-2fa5ad62b841-kube-api-access-6tfw9\") pod \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\" (UID: \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\") "
Nov 03 11:31:08.636667 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.636576    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a3de92-9886-41bd-99ab-2fa5ad62b841-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a3de92-9886-41bd-99ab-2fa5ad62b841-plugins-config-volume\") pod \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\" (UID: \"f3a3de92-9886-41bd-99ab-2fa5ad62b841\") "
Nov 03 11:31:08.636972 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.636950    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a3de92-9886-41bd-99ab-2fa5ad62b841-plugins-config-volume" pod "f3a3de92-9886-41bd-99ab-2fa5ad62b841" (UID: "f3a3de92-9886-41bd-99ab-2fa5ad62b841"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:08.637127 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.637113    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a3de92-9886-41bd-99ab-2fa5ad62b841-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:08.639240 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.639199    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a3de92-9886-41bd-99ab-2fa5ad62b841-kube-api-access-6tfw9" pod "f3a3de92-9886-41bd-99ab-2fa5ad62b841" (UID: "f3a3de92-9886-41bd-99ab-2fa5ad62b841"). InnerVolumeSpecName "kube-api-access-6tfw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:08.738415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.738376    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b087ec3a-d711-4e4f-b219-c6d026fa08c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b087ec3a-d711-4e4f-b219-c6d026fa08c4-plugins-config-volume\") pod \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\" (UID: \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\") "
Nov 03 11:31:08.738587 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.738502    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b087ec3a-d711-4e4f-b219-c6d026fa08c4-kube-api-access-q56qq\" (UniqueName: \"kubernetes.io/projected/b087ec3a-d711-4e4f-b219-c6d026fa08c4-kube-api-access-q56qq\") pod \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\" (UID: \"b087ec3a-d711-4e4f-b219-c6d026fa08c4\") "
Nov 03 11:31:08.738753 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.738732    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b087ec3a-d711-4e4f-b219-c6d026fa08c4-plugins-config-volume" pod "b087ec3a-d711-4e4f-b219-c6d026fa08c4" (UID: "b087ec3a-d711-4e4f-b219-c6d026fa08c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:08.738995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.738980    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b087ec3a-d711-4e4f-b219-c6d026fa08c4-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:08.739039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.738998    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfw9\" (UniqueName: \"kubernetes.io/projected/f3a3de92-9886-41bd-99ab-2fa5ad62b841-kube-api-access-6tfw9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:08.741305 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.741281    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b087ec3a-d711-4e4f-b219-c6d026fa08c4-kube-api-access-q56qq" pod "b087ec3a-d711-4e4f-b219-c6d026fa08c4" (UID: "b087ec3a-d711-4e4f-b219-c6d026fa08c4"). InnerVolumeSpecName "kube-api-access-q56qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:08.840229 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:08.840155    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q56qq\" (UniqueName: \"kubernetes.io/projected/b087ec3a-d711-4e4f-b219-c6d026fa08c4-kube-api-access-q56qq\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:09.302354 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.302321    5393 generic.go:356] "Generic (PLEG): container finished" podID="b087ec3a-d711-4e4f-b219-c6d026fa08c4" containerID="ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4" exitCode=0
Nov 03 11:31:09.302550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.302382    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"
Nov 03 11:31:09.302550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.302404    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk" event={"ID":"b087ec3a-d711-4e4f-b219-c6d026fa08c4","Type":"ContainerDied","Data":"ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4"}
Nov 03 11:31:09.302550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.302455    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk" event={"ID":"b087ec3a-d711-4e4f-b219-c6d026fa08c4","Type":"ContainerDied","Data":"6fad796d6bb7056a7fa7b57b367019cf861cc8f34d8ad9b7cc8fbcd77785c4c3"}
Nov 03 11:31:09.302550 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.302482    5393 scope.go:122] "RemoveContainer" containerID="ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4"
Nov 03 11:31:09.304444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.304419    5393 generic.go:356] "Generic (PLEG): container finished" podID="f3a3de92-9886-41bd-99ab-2fa5ad62b841" containerID="f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d" exitCode=0
Nov 03 11:31:09.304575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.304473    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"
Nov 03 11:31:09.304575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.304504    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws" event={"ID":"f3a3de92-9886-41bd-99ab-2fa5ad62b841","Type":"ContainerDied","Data":"f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d"}
Nov 03 11:31:09.304575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.304554    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws" event={"ID":"f3a3de92-9886-41bd-99ab-2fa5ad62b841","Type":"ContainerDied","Data":"24cf7ae75382ef73d80b99af46a649d5452373e83dc4c70809e078e7deeaaa6a"}
Nov 03 11:31:09.313711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.313691    5393 scope.go:122] "RemoveContainer" containerID="ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4"
Nov 03 11:31:09.314045 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:09.314022    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4\": container with ID starting with ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4 not found: ID does not exist" containerID="ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4"
Nov 03 11:31:09.314160 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.314054    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4"} err="failed to get container status \"ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4\": rpc error: code = NotFound desc = could not find container \"ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4\": container with ID starting with ab2277d5bd4d8e81f7f6552ffad505db0194a9cfc67477c9f21150cc2a2adde4 not found: ID does not exist"
Nov 03 11:31:09.314160 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.314074    5393 scope.go:122] "RemoveContainer" containerID="f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d"
Nov 03 11:31:09.324221 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.324180    5393 scope.go:122] "RemoveContainer" containerID="f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d"
Nov 03 11:31:09.325154 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:09.325119    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d\": container with ID starting with f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d not found: ID does not exist" containerID="f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d"
Nov 03 11:31:09.325154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.325146    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d"} err="failed to get container status \"f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d\": rpc error: code = NotFound desc = could not find container \"f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d\": container with ID starting with f2518843c8ebdeefdb0c8ca8a413c8922c4229e887730c59fafcc86ddec7266d not found: ID does not exist"
Nov 03 11:31:09.328339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.328316    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"]
Nov 03 11:31:09.334441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.334405    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-56687c-54bf7c6bf5-lnbqk"]
Nov 03 11:31:09.344387 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.344294    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"]
Nov 03 11:31:09.347735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:09.347711    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-56687c-85989ddcb7-5zjws"]
Nov 03 11:31:10.235403 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:10.235366    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b087ec3a-d711-4e4f-b219-c6d026fa08c4" path="/var/lib/kubelet/pods/b087ec3a-d711-4e4f-b219-c6d026fa08c4/volumes"
Nov 03 11:31:10.236022 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:10.235998    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a3de92-9886-41bd-99ab-2fa5ad62b841" path="/var/lib/kubelet/pods/f3a3de92-9886-41bd-99ab-2fa5ad62b841/volumes"
Nov 03 11:31:13.189901 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:13.189863    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:31:15.494875 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:15.494838    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"]
Nov 03 11:31:15.502321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:15.502290    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-9c53b2-jrm4x"]
Nov 03 11:31:16.231262 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:16.231227    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a17c39-ad43-4a41-83c3-eca3245321f8" path="/var/lib/kubelet/pods/d2a17c39-ad43-4a41-83c3-eca3245321f8/volumes"
Nov 03 11:31:17.424219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:17.424182    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"]
Nov 03 11:31:20.064872 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.064827    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q" podUID="f572826e-378d-4514-9055-e68f3872b0ea" containerName="main" containerID="cri-o://d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b" gracePeriod=45
Nov 03 11:31:20.381677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.373219    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:31:20.437089 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437059    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-dshm\") pod \"f572826e-378d-4514-9055-e68f3872b0ea\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") "
Nov 03 11:31:20.437269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437100    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-home\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-home\") pod \"f572826e-378d-4514-9055-e68f3872b0ea\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") "
Nov 03 11:31:20.437269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437170    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-model-cache\") pod \"f572826e-378d-4514-9055-e68f3872b0ea\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") "
Nov 03 11:31:20.437269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437237    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-tmp-dir\") pod \"f572826e-378d-4514-9055-e68f3872b0ea\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") "
Nov 03 11:31:20.437269 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437269    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f572826e-378d-4514-9055-e68f3872b0ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/f572826e-378d-4514-9055-e68f3872b0ea-tls-certs\") pod \"f572826e-378d-4514-9055-e68f3872b0ea\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") "
Nov 03 11:31:20.437509 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437358    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f572826e-378d-4514-9055-e68f3872b0ea-kube-api-access-sqzmn\" (UniqueName: \"kubernetes.io/projected/f572826e-378d-4514-9055-e68f3872b0ea-kube-api-access-sqzmn\") pod \"f572826e-378d-4514-9055-e68f3872b0ea\" (UID: \"f572826e-378d-4514-9055-e68f3872b0ea\") "
Nov 03 11:31:20.437509 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437388    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-home" pod "f572826e-378d-4514-9055-e68f3872b0ea" (UID: "f572826e-378d-4514-9055-e68f3872b0ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:20.437612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437541    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-tmp-dir" pod "f572826e-378d-4514-9055-e68f3872b0ea" (UID: "f572826e-378d-4514-9055-e68f3872b0ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:20.437612 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437554    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-model-cache" pod "f572826e-378d-4514-9055-e68f3872b0ea" (UID: "f572826e-378d-4514-9055-e68f3872b0ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:20.437867 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437843    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:20.437867 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437869    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:20.438123 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.437883    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:20.439620 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.439597    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f572826e-378d-4514-9055-e68f3872b0ea-tls-certs" pod "f572826e-378d-4514-9055-e68f3872b0ea" (UID: "f572826e-378d-4514-9055-e68f3872b0ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:20.439735 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.439618    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-dshm" pod "f572826e-378d-4514-9055-e68f3872b0ea" (UID: "f572826e-378d-4514-9055-e68f3872b0ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:20.439783 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.439741    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f572826e-378d-4514-9055-e68f3872b0ea-kube-api-access-sqzmn" pod "f572826e-378d-4514-9055-e68f3872b0ea" (UID: "f572826e-378d-4514-9055-e68f3872b0ea"). InnerVolumeSpecName "kube-api-access-sqzmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:20.466087 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.466052    5393 generic.go:356] "Generic (PLEG): container finished" podID="f572826e-378d-4514-9055-e68f3872b0ea" containerID="d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b" exitCode=0
Nov 03 11:31:20.466237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.466114    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"
Nov 03 11:31:20.466237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.466166    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q" event={"ID":"f572826e-378d-4514-9055-e68f3872b0ea","Type":"ContainerDied","Data":"d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b"}
Nov 03 11:31:20.466237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.466213    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q" event={"ID":"f572826e-378d-4514-9055-e68f3872b0ea","Type":"ContainerDied","Data":"dad246baf49bcd289fbba578ccf7832eac7e6f0a3001d064d9da7ee4e0fa4a4d"}
Nov 03 11:31:20.466237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.466236    5393 scope.go:122] "RemoveContainer" containerID="d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b"
Nov 03 11:31:20.479289 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.479267    5393 scope.go:122] "RemoveContainer" containerID="d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b"
Nov 03 11:31:20.479600 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:20.479579    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b\": container with ID starting with d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b not found: ID does not exist" containerID="d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b"
Nov 03 11:31:20.479688 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.479611    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b"} err="failed to get container status \"d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b\": rpc error: code = NotFound desc = could not find container \"d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b\": container with ID starting with d0fb0aa30158970d714783ded521e1f318aee630684e97c013d86dc3a1cfec1b not found: ID does not exist"
Nov 03 11:31:20.493621 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.493564    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"]
Nov 03 11:31:20.496865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.496843    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-6012b5/rate-test-model-6012b5-kserve-7fcbcb94d5-cch8q"]
Nov 03 11:31:20.539393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.539358    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f572826e-378d-4514-9055-e68f3872b0ea-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:20.539574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.539427    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqzmn\" (UniqueName: \"kubernetes.io/projected/f572826e-378d-4514-9055-e68f3872b0ea-kube-api-access-sqzmn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:20.539574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:20.539444    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f572826e-378d-4514-9055-e68f3872b0ea-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:22.149404 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.149365    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd" podUID="0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" containerName="main" containerID="cri-o://3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517" gracePeriod=45
Nov 03 11:31:22.231237 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.231197    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f572826e-378d-4514-9055-e68f3872b0ea" path="/var/lib/kubelet/pods/f572826e-378d-4514-9055-e68f3872b0ea/volumes"
Nov 03 11:31:22.425435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.425344    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t" podUID="b2aa0412-2a7f-4850-ba77-92f35382670f" containerName="payload-processing" containerID="cri-o://64700768101b9dfc287508e08ff3e164baf2f75080c663f0f7a68b65892aaf27" gracePeriod=25
Nov 03 11:31:22.452696 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.452670    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:31:22.481548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.481311    5393 generic.go:356] "Generic (PLEG): container finished" podID="0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" containerID="3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517" exitCode=0
Nov 03 11:31:22.481548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.481369    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd" event={"ID":"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae","Type":"ContainerDied","Data":"3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517"}
Nov 03 11:31:22.481548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.481423    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd" event={"ID":"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae","Type":"ContainerDied","Data":"bc154c65ae2d0a57e69efa4ceeae1df021b5764f8fd43129f34a044fef8fa7b2"}
Nov 03 11:31:22.481548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.481375    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"
Nov 03 11:31:22.481548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.481457    5393 scope.go:122] "RemoveContainer" containerID="3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517"
Nov 03 11:31:22.493488 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.493462    5393 scope.go:122] "RemoveContainer" containerID="3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517"
Nov 03 11:31:22.493867 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:22.493828    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517\": container with ID starting with 3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517 not found: ID does not exist" containerID="3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517"
Nov 03 11:31:22.493969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.493881    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517"} err="failed to get container status \"3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517\": rpc error: code = NotFound desc = could not find container \"3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517\": container with ID starting with 3bf1b9c6ae2cc92130c76ad421ef777ed1f9fc1dce13165902a65de204b15517 not found: ID does not exist"
Nov 03 11:31:22.584716 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.584678    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tls-certs\") pod \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") "
Nov 03 11:31:22.584962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.584741    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-home\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-home\") pod \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") "
Nov 03 11:31:22.584962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.584767    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-dshm\") pod \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") "
Nov 03 11:31:22.584962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.584890    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tmp-dir\") pod \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") "
Nov 03 11:31:22.585161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.584979    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-model-cache\") pod \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") "
Nov 03 11:31:22.585161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.585014    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-kube-api-access-dncnl\" (UniqueName: \"kubernetes.io/projected/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-kube-api-access-dncnl\") pod \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\" (UID: \"0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae\") "
Nov 03 11:31:22.585161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.585096    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-home" pod "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" (UID: "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:22.585386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.585338    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-model-cache" pod "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" (UID: "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:22.585386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.585351    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tmp-dir" pod "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" (UID: "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:22.585699 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.585677    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:22.585780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.585705    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:22.585780 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.585718    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:22.587202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.587178    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tls-certs" pod "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" (UID: "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:22.587362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.587349    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-kube-api-access-dncnl" pod "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" (UID: "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae"). InnerVolumeSpecName "kube-api-access-dncnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:22.587476 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.587461    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-dshm" pod "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" (UID: "0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:22.687448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.687360    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:22.687448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.687392    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dncnl\" (UniqueName: \"kubernetes.io/projected/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-kube-api-access-dncnl\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:22.687448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.687402    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:22.805384 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.805355    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"]
Nov 03 11:31:22.808770 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:22.808742    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cc8214/rate-test-model-cc8214-kserve-58668ff977-4r4qd"]
Nov 03 11:31:23.268038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.267993    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"]
Nov 03 11:31:23.268596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.268293    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy" containerID="cri-o://0134ce9ac2cc24c9414b5a35d8713b10f7dc256341d90c596098234a95c07326" gracePeriod=30
Nov 03 11:31:23.979293 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979256    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"]
Nov 03 11:31:23.979858 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979843    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f572826e-378d-4514-9055-e68f3872b0ea" containerName="main"
Nov 03 11:31:23.979906 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979860    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f572826e-378d-4514-9055-e68f3872b0ea" containerName="main"
Nov 03 11:31:23.979906 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979883    5393 cpu_manager.go:414] "containerMap: removing container" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy"
Nov 03 11:31:23.979906 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979889    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy"
Nov 03 11:31:23.979906 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979898    5393 cpu_manager.go:414] "containerMap: removing container" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:31:23.979906 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979903    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979931    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" containerName="main"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979936    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" containerName="main"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979957    5393 cpu_manager.go:414] "containerMap: removing container" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979963    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979979    5393 cpu_manager.go:414] "containerMap: removing container" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979983    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639d6260-0fdd-4698-83bc-61060581644f" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979991    5393 cpu_manager.go:414] "containerMap: removing container" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.979995    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7171db8f-769b-4dba-9e97-31b1a5ed949d" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980002    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f3a3de92-9886-41bd-99ab-2fa5ad62b841" containerName="payload-processing"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980006    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a3de92-9886-41bd-99ab-2fa5ad62b841" containerName="payload-processing"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980016    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980020    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980032    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b087ec3a-d711-4e4f-b219-c6d026fa08c4" containerName="payload-pre-processing"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980039    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b087ec3a-d711-4e4f-b219-c6d026fa08c4" containerName="payload-pre-processing"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980053    5393 cpu_manager.go:414] "containerMap: removing container" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980059    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980069    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d2a17c39-ad43-4a41-83c3-eca3245321f8" containerName="revoke-keys"
Nov 03 11:31:23.980102 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980075    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a17c39-ad43-4a41-83c3-eca3245321f8" containerName="revoke-keys"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980156    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="799b4527-5c0d-40c7-9e8b-03417efc5d9c" containerName="cleanup"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980166    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" containerName="main"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980174    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980180    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ff459d-34b6-440e-afe0-061e1d7860a9" containerName="cleanup"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980188    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a3de92-9886-41bd-99ab-2fa5ad62b841" containerName="payload-processing"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980197    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="049a4cb6-866c-4d73-a260-7383b3fa0891" containerName="cleanup"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980205    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9471f345-99b5-4ca1-b2fb-1995db880fda" containerName="cleanup"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980211    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b087ec3a-d711-4e4f-b219-c6d026fa08c4" containerName="payload-pre-processing"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980218    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e50e5e-7b9a-4a24-9def-949fed6aca68" containerName="cleanup"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980223    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a17c39-ad43-4a41-83c3-eca3245321f8" containerName="revoke-keys"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980228    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f572826e-378d-4514-9055-e68f3872b0ea" containerName="main"
Nov 03 11:31:23.980692 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.980234    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a9215b-e2bd-4992-a6d6-97add590eb3b" containerName="istio-proxy"
Nov 03 11:31:23.984653 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.984633    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:23.989626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.989602    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:31:23.990026 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.990005    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:31:23.990799 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:23.990577    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"]
Nov 03 11:31:24.118567 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.118529    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhm8b\" (UniqueName: \"kubernetes.io/projected/75b67e08-c92f-415c-a970-d9bb0a67eb79-kube-api-access-nhm8b\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.118771 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.118661    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.118830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.118773    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.220249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.220212    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.220456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.220320    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm8b\" (UniqueName: \"kubernetes.io/projected/75b67e08-c92f-415c-a970-d9bb0a67eb79-kube-api-access-nhm8b\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.220456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.220390    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.220826 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.220806    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.223043 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.223022    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.228793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.228765    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhm8b\" (UniqueName: \"kubernetes.io/projected/75b67e08-c92f-415c-a970-d9bb0a67eb79-kube-api-access-nhm8b\") pod \"maas-api-revoke-sub-dc0fdf6e-bs96f\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.230831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.230771    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae" path="/var/lib/kubelet/pods/0c1c1135-6b3a-4b96-b0cc-d0a6eb07d3ae/volumes"
Nov 03 11:31:24.305114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.305078    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:24.587014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:24.586976    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"]
Nov 03 11:31:24.588122 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:31:24.588098    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b67e08_c92f_415c_a970_d9bb0a67eb79.slice/crio-70c4bcf809e0089f8a67d55eb578d4ae8de2514e5d66a59fb777cc8713d5d730 WatchSource:0}: Error finding container 70c4bcf809e0089f8a67d55eb578d4ae8de2514e5d66a59fb777cc8713d5d730: Status 404 returned error can't find the container with id 70c4bcf809e0089f8a67d55eb578d4ae8de2514e5d66a59fb777cc8713d5d730
Nov 03 11:31:25.505401 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:25.505360    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f" event={"ID":"75b67e08-c92f-415c-a970-d9bb0a67eb79","Type":"ContainerStarted","Data":"7331466de964ac2af72d03437a8f34fa7325333ba18a9ffb370e976ffa63e99c"}
Nov 03 11:31:25.505860 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:25.505418    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f" event={"ID":"75b67e08-c92f-415c-a970-d9bb0a67eb79","Type":"ContainerStarted","Data":"70c4bcf809e0089f8a67d55eb578d4ae8de2514e5d66a59fb777cc8713d5d730"}
Nov 03 11:31:25.524830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:25.524775    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f" podStartSLOduration=2.524759758 podStartE2EDuration="2.524759758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:31:25.522446101 +0000 UTC m=+3729.427300542" watchObservedRunningTime="2026-11-03 11:31:25.524759758 +0000 UTC m=+3729.429614199"
Nov 03 11:31:26.318874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:26.318787    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"]
Nov 03 11:31:26.319144 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:26.319112    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy" containerID="cri-o://133e978b3f1dc3013feb07c7fe9656964f1bea2035660f04b6824cb97750e860" gracePeriod=30
Nov 03 11:31:26.515978 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:26.515935    5393 generic.go:356] "Generic (PLEG): container finished" podID="75b67e08-c92f-415c-a970-d9bb0a67eb79" containerID="7331466de964ac2af72d03437a8f34fa7325333ba18a9ffb370e976ffa63e99c" exitCode=0
Nov 03 11:31:26.516533 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:26.516033    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f" event={"ID":"75b67e08-c92f-415c-a970-d9bb0a67eb79","Type":"ContainerDied","Data":"7331466de964ac2af72d03437a8f34fa7325333ba18a9ffb370e976ffa63e99c"}
Nov 03 11:31:27.674664 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.674639    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:27.779452 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.779362    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b67e08-c92f-415c-a970-d9bb0a67eb79-kube-api-access-nhm8b\" (UniqueName: \"kubernetes.io/projected/75b67e08-c92f-415c-a970-d9bb0a67eb79-kube-api-access-nhm8b\") pod \"75b67e08-c92f-415c-a970-d9bb0a67eb79\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") "
Nov 03 11:31:27.779656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.779495    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-xks-ca\") pod \"75b67e08-c92f-415c-a970-d9bb0a67eb79\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") "
Nov 03 11:31:27.779767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.779746    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-openshift-ca\") pod \"75b67e08-c92f-415c-a970-d9bb0a67eb79\" (UID: \"75b67e08-c92f-415c-a970-d9bb0a67eb79\") "
Nov 03 11:31:27.780224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.780132    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-openshift-ca" pod "75b67e08-c92f-415c-a970-d9bb0a67eb79" (UID: "75b67e08-c92f-415c-a970-d9bb0a67eb79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:27.780359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.780276    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:27.781650 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.781626    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-xks-ca" pod "75b67e08-c92f-415c-a970-d9bb0a67eb79" (UID: "75b67e08-c92f-415c-a970-d9bb0a67eb79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:27.781787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.781768    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b67e08-c92f-415c-a970-d9bb0a67eb79-kube-api-access-nhm8b" pod "75b67e08-c92f-415c-a970-d9bb0a67eb79" (UID: "75b67e08-c92f-415c-a970-d9bb0a67eb79"). InnerVolumeSpecName "kube-api-access-nhm8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:27.882077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.882040    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhm8b\" (UniqueName: \"kubernetes.io/projected/75b67e08-c92f-415c-a970-d9bb0a67eb79-kube-api-access-nhm8b\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:27.882077 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:27.882070    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b67e08-c92f-415c-a970-d9bb0a67eb79-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.534647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.534454    5393 generic.go:356] "Generic (PLEG): container finished" podID="3761a592-894d-4ce7-af27-594c7498dc45" containerID="0134ce9ac2cc24c9414b5a35d8713b10f7dc256341d90c596098234a95c07326" exitCode=0
Nov 03 11:31:28.534647 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.534521    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" event={"ID":"3761a592-894d-4ce7-af27-594c7498dc45","Type":"ContainerDied","Data":"0134ce9ac2cc24c9414b5a35d8713b10f7dc256341d90c596098234a95c07326"}
Nov 03 11:31:28.536373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.536353    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f"
Nov 03 11:31:28.536373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.536360    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc0fdf6e-bs96f" event={"ID":"75b67e08-c92f-415c-a970-d9bb0a67eb79","Type":"ContainerDied","Data":"70c4bcf809e0089f8a67d55eb578d4ae8de2514e5d66a59fb777cc8713d5d730"}
Nov 03 11:31:28.536557 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.536405    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c4bcf809e0089f8a67d55eb578d4ae8de2514e5d66a59fb777cc8713d5d730"
Nov 03 11:31:28.540766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.540749    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:31:28.692266 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692234    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-envoy\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.692703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692302    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3761a592-894d-4ce7-af27-594c7498dc45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3761a592-894d-4ce7-af27-594c7498dc45-istio-podinfo\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.692703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692343    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-data\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.692703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692510    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-socket\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.692703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692575    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3761a592-894d-4ce7-af27-594c7498dc45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3761a592-894d-4ce7-af27-594c7498dc45-istiod-ca-cert\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.692703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692609    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-kube-api-access-zm42w\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-kube-api-access-zm42w\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.692703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692693    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-certs\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.693090 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692779    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-credential-socket\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.693090 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692690    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-data" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.693090 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692776    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-socket" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.693090 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692886    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-istio-token\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-istio-token\") pod \"3761a592-894d-4ce7-af27-594c7498dc45\" (UID: \"3761a592-894d-4ce7-af27-594c7498dc45\") "
Nov 03 11:31:28.693090 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.692987    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-certs" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.693090 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.693075    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-credential-socket" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.693334 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.693099    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3761a592-894d-4ce7-af27-594c7498dc45-istiod-ca-cert" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:28.693763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.693703    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3761a592-894d-4ce7-af27-594c7498dc45-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.693763 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.693759    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.693974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.693778    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.693974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.693794    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.693974 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.693808    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.694908 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.694879    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3761a592-894d-4ce7-af27-594c7498dc45-istio-podinfo" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:31:28.695041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.695007    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-kube-api-access-zm42w" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "kube-api-access-zm42w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:28.695311 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.695296    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-istio-token" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:28.695502 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.695488    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-envoy" pod "3761a592-894d-4ce7-af27-594c7498dc45" (UID: "3761a592-894d-4ce7-af27-594c7498dc45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:28.795623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.795536    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3761a592-894d-4ce7-af27-594c7498dc45-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.795623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.795567    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3761a592-894d-4ce7-af27-594c7498dc45-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.795623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.795577    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm42w\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-kube-api-access-zm42w\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.795623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.795587    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3761a592-894d-4ce7-af27-594c7498dc45-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:28.806177 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:28.806135    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t" podUID="b2aa0412-2a7f-4850-ba77-92f35382670f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:29.544310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:29.544276    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"
Nov 03 11:31:29.544310 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:29.544292    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp" event={"ID":"3761a592-894d-4ce7-af27-594c7498dc45","Type":"ContainerDied","Data":"5eab6dd9c929930bd92632c2aca3b1560f11aafece2594c828dd3d24212fb8c1"}
Nov 03 11:31:29.544554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:29.544347    5393 scope.go:122] "RemoveContainer" containerID="0134ce9ac2cc24c9414b5a35d8713b10f7dc256341d90c596098234a95c07326"
Nov 03 11:31:29.569374 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:29.569350    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"]
Nov 03 11:31:29.573850 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:29.573829    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-9c53b2-openshift-default-8474f7657f-qd7jp"]
Nov 03 11:31:30.230792 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:30.230758    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3761a592-894d-4ce7-af27-594c7498dc45" path="/var/lib/kubelet/pods/3761a592-894d-4ce7-af27-594c7498dc45/volumes"
Nov 03 11:31:31.574626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.574299    5393 generic.go:356] "Generic (PLEG): container finished" podID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerID="133e978b3f1dc3013feb07c7fe9656964f1bea2035660f04b6824cb97750e860" exitCode=0
Nov 03 11:31:31.574626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.574451    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" event={"ID":"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5","Type":"ContainerDied","Data":"133e978b3f1dc3013feb07c7fe9656964f1bea2035660f04b6824cb97750e860"}
Nov 03 11:31:31.574626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.574512    5393 scope.go:122] "RemoveContainer" containerID="004134a50ed4b5e5d0e2b8d95468203ce976a3411605a9b0135306af0b8eeec6"
Nov 03 11:31:31.623362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.623335    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:31:31.734596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.734561    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-kube-api-access-54s6n\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-kube-api-access-54s6n\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.734755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.734625    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-certs\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.734755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.734678    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-podinfo\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.734755 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.734741    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-socket\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.734893 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.734814    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-envoy\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.734893 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.734844    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-credential-socket\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.734893 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.734878    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-data\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.735112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.734978    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istiod-ca-cert\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.735112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735010    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-certs" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:31.735112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735050    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-token\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-token\") pod \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\" (UID: \"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5\") "
Nov 03 11:31:31.735336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735047    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-socket" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:31.735336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735203    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-credential-socket" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:31.735336 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735270    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-data" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:31.735781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735422    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istiod-ca-cert" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:31.735781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735622    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.735781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735638    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.735781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735651    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.735781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735660    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.735781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.735671    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.737131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.737102    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-kube-api-access-54s6n" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "kube-api-access-54s6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:31.737228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.737176    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-podinfo" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:31:31.737729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.737704    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-token" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:31.737817 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.737772    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-envoy" pod "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" (UID: "432522bc-6cbe-4cb5-a30c-9f9fad1b6be5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:31.836835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.836796    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.836835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.836829    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54s6n\" (UniqueName: \"kubernetes.io/projected/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-kube-api-access-54s6n\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.836835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.836839    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:31.837101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:31.836848    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:32.583781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:32.583701    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"
Nov 03 11:31:32.583781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:32.583722    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc" event={"ID":"432522bc-6cbe-4cb5-a30c-9f9fad1b6be5","Type":"ContainerDied","Data":"162c9806e9afe92a625aaa6e6c4075219831caf0d8b74048655b9b6f7c1b5e3e"}
Nov 03 11:31:32.583781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:32.583782    5393 scope.go:122] "RemoveContainer" containerID="133e978b3f1dc3013feb07c7fe9656964f1bea2035660f04b6824cb97750e860"
Nov 03 11:31:32.603526 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:32.603496    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"]
Nov 03 11:31:32.607158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:32.607134    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-917da2-openshift-default-578f45548f-mzzcc"]
Nov 03 11:31:34.230600 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:34.230568    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" path="/var/lib/kubelet/pods/432522bc-6cbe-4cb5-a30c-9f9fad1b6be5/volumes"
Nov 03 11:31:36.849368 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.849333    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"]
Nov 03 11:31:36.849957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.849936    5393 cpu_manager.go:414] "containerMap: removing container" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy"
Nov 03 11:31:36.850025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.849960    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy"
Nov 03 11:31:36.850025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.849975    5393 cpu_manager.go:414] "containerMap: removing container" podUID="75b67e08-c92f-415c-a970-d9bb0a67eb79" containerName="revoke-keys"
Nov 03 11:31:36.850025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.849982    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b67e08-c92f-415c-a970-d9bb0a67eb79" containerName="revoke-keys"
Nov 03 11:31:36.850025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.849994    5393 cpu_manager.go:414] "containerMap: removing container" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy"
Nov 03 11:31:36.850025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.850002    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy"
Nov 03 11:31:36.850025 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.850024    5393 cpu_manager.go:414] "containerMap: removing container" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy"
Nov 03 11:31:36.850224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.850029    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy"
Nov 03 11:31:36.850224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.850140    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy"
Nov 03 11:31:36.850224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.850152    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b67e08-c92f-415c-a970-d9bb0a67eb79" containerName="revoke-keys"
Nov 03 11:31:36.850224 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.850162    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3761a592-894d-4ce7-af27-594c7498dc45" containerName="istio-proxy"
Nov 03 11:31:36.854945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.854894    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:36.859513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.859475    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"]
Nov 03 11:31:36.859677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.859546    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:31:36.859677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.859626    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:31:36.898166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.898130    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntf7\" (UniqueName: \"kubernetes.io/projected/687ae590-f4c4-4aec-97b1-bb60e70299bd-kube-api-access-5ntf7\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:36.898318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.898296    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:36.898486 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.898459    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:36.999711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.999674    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntf7\" (UniqueName: \"kubernetes.io/projected/687ae590-f4c4-4aec-97b1-bb60e70299bd-kube-api-access-5ntf7\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:36.999964 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.999820    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:37.000049 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:36.999977    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:37.000613 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:37.000594    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:37.002427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:37.002404    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:37.014308 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:37.014281    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntf7\" (UniqueName: \"kubernetes.io/projected/687ae590-f4c4-4aec-97b1-bb60e70299bd-kube-api-access-5ntf7\") pod \"maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:37.170477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:37.170386    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:37.462296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:37.462269    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"]
Nov 03 11:31:37.463546 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:31:37.463516    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687ae590_f4c4_4aec_97b1_bb60e70299bd.slice/crio-d6f9357b05c8235c60b362f4bf26ab6ac677231af983a2961ff7de8ed4fc23b7 WatchSource:0}: Error finding container d6f9357b05c8235c60b362f4bf26ab6ac677231af983a2961ff7de8ed4fc23b7: Status 404 returned error can't find the container with id d6f9357b05c8235c60b362f4bf26ab6ac677231af983a2961ff7de8ed4fc23b7
Nov 03 11:31:37.620339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:37.620290    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz" event={"ID":"687ae590-f4c4-4aec-97b1-bb60e70299bd","Type":"ContainerStarted","Data":"d6f9357b05c8235c60b362f4bf26ab6ac677231af983a2961ff7de8ed4fc23b7"}
Nov 03 11:31:38.630576 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:38.630542    5393 generic.go:356] "Generic (PLEG): container finished" podID="687ae590-f4c4-4aec-97b1-bb60e70299bd" containerID="b66c1c20ef638735c6c3d267e19322fda0ac64165fd4a1a37b5b7820c1639edb" exitCode=0
Nov 03 11:31:38.631029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:38.630630    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz" event={"ID":"687ae590-f4c4-4aec-97b1-bb60e70299bd","Type":"ContainerDied","Data":"b66c1c20ef638735c6c3d267e19322fda0ac64165fd4a1a37b5b7820c1639edb"}
Nov 03 11:31:38.806581 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:38.806486    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t" podUID="b2aa0412-2a7f-4850-ba77-92f35382670f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:39.788298 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.788271    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:39.834390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.834359    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-xks-ca\") pod \"687ae590-f4c4-4aec-97b1-bb60e70299bd\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") "
Nov 03 11:31:39.834598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.834567    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687ae590-f4c4-4aec-97b1-bb60e70299bd-kube-api-access-5ntf7\" (UniqueName: \"kubernetes.io/projected/687ae590-f4c4-4aec-97b1-bb60e70299bd-kube-api-access-5ntf7\") pod \"687ae590-f4c4-4aec-97b1-bb60e70299bd\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") "
Nov 03 11:31:39.834598 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.834590    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-openshift-ca\") pod \"687ae590-f4c4-4aec-97b1-bb60e70299bd\" (UID: \"687ae590-f4c4-4aec-97b1-bb60e70299bd\") "
Nov 03 11:31:39.835032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.835008    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-openshift-ca" pod "687ae590-f4c4-4aec-97b1-bb60e70299bd" (UID: "687ae590-f4c4-4aec-97b1-bb60e70299bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:39.837016 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.836988    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687ae590-f4c4-4aec-97b1-bb60e70299bd-kube-api-access-5ntf7" pod "687ae590-f4c4-4aec-97b1-bb60e70299bd" (UID: "687ae590-f4c4-4aec-97b1-bb60e70299bd"). InnerVolumeSpecName "kube-api-access-5ntf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:39.837163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.837146    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-xks-ca" pod "687ae590-f4c4-4aec-97b1-bb60e70299bd" (UID: "687ae590-f4c4-4aec-97b1-bb60e70299bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:31:39.936658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.936573    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntf7\" (UniqueName: \"kubernetes.io/projected/687ae590-f4c4-4aec-97b1-bb60e70299bd-kube-api-access-5ntf7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:39.936658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.936603    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:39.936658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:39.936615    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687ae590-f4c4-4aec-97b1-bb60e70299bd-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:40.645116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:40.645086    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"
Nov 03 11:31:40.645116 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:40.645092    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz" event={"ID":"687ae590-f4c4-4aec-97b1-bb60e70299bd","Type":"ContainerDied","Data":"d6f9357b05c8235c60b362f4bf26ab6ac677231af983a2961ff7de8ed4fc23b7"}
Nov 03 11:31:40.645339 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:40.645148    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6f9357b05c8235c60b362f4bf26ab6ac677231af983a2961ff7de8ed4fc23b7"
Nov 03 11:31:42.672781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.672746    5393 generic.go:356] "Generic (PLEG): container finished" podID="b2aa0412-2a7f-4850-ba77-92f35382670f" containerID="64700768101b9dfc287508e08ff3e164baf2f75080c663f0f7a68b65892aaf27" exitCode=0
Nov 03 11:31:42.673152 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.672820    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t" event={"ID":"b2aa0412-2a7f-4850-ba77-92f35382670f","Type":"ContainerDied","Data":"64700768101b9dfc287508e08ff3e164baf2f75080c663f0f7a68b65892aaf27"}
Nov 03 11:31:42.690446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.690379    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:31:42.788946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.788866    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2aa0412-2a7f-4850-ba77-92f35382670f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2aa0412-2a7f-4850-ba77-92f35382670f-plugins-config-volume\") pod \"b2aa0412-2a7f-4850-ba77-92f35382670f\" (UID: \"b2aa0412-2a7f-4850-ba77-92f35382670f\") "
Nov 03 11:31:42.789172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.788986    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2aa0412-2a7f-4850-ba77-92f35382670f-kube-api-access-tfhjs\" (UniqueName: \"kubernetes.io/projected/b2aa0412-2a7f-4850-ba77-92f35382670f-kube-api-access-tfhjs\") pod \"b2aa0412-2a7f-4850-ba77-92f35382670f\" (UID: \"b2aa0412-2a7f-4850-ba77-92f35382670f\") "
Nov 03 11:31:42.789593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.789344    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2aa0412-2a7f-4850-ba77-92f35382670f-plugins-config-volume" pod "b2aa0412-2a7f-4850-ba77-92f35382670f" (UID: "b2aa0412-2a7f-4850-ba77-92f35382670f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:42.789768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.789745    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2aa0412-2a7f-4850-ba77-92f35382670f-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:42.792418 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.792389    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2aa0412-2a7f-4850-ba77-92f35382670f-kube-api-access-tfhjs" pod "b2aa0412-2a7f-4850-ba77-92f35382670f" (UID: "b2aa0412-2a7f-4850-ba77-92f35382670f"). InnerVolumeSpecName "kube-api-access-tfhjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:42.891241 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:42.891209    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhjs\" (UniqueName: \"kubernetes.io/projected/b2aa0412-2a7f-4850-ba77-92f35382670f-kube-api-access-tfhjs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:43.190966 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:43.190902    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:31:43.689196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:43.689153    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t" event={"ID":"b2aa0412-2a7f-4850-ba77-92f35382670f","Type":"ContainerDied","Data":"ace17dcc5f014d5284f21abd40316d030f2a4cd3b6ce435b91dd794eb9b460aa"}
Nov 03 11:31:43.689704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:43.689209    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"
Nov 03 11:31:43.689704 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:43.689263    5393 scope.go:122] "RemoveContainer" containerID="64700768101b9dfc287508e08ff3e164baf2f75080c663f0f7a68b65892aaf27"
Nov 03 11:31:43.712433 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:43.712403    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"]
Nov 03 11:31:43.716998 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:43.716971    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-917da2-5c9fcf7f59-cgr7t"]
Nov 03 11:31:44.229751 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:44.229701    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2aa0412-2a7f-4850-ba77-92f35382670f" path="/var/lib/kubelet/pods/b2aa0412-2a7f-4850-ba77-92f35382670f/volumes"
Nov 03 11:31:46.429505 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:46.429465    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"]
Nov 03 11:31:46.430205 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:46.429815    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy" containerID="cri-o://a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6" gracePeriod=30
Nov 03 11:31:46.973350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:46.973267    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"]
Nov 03 11:31:47.008837 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:47.008797    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"]
Nov 03 11:31:48.115462 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:48.115427    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"]
Nov 03 11:31:48.118101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:48.118076    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fb5766-p7zzz"]
Nov 03 11:31:48.230029 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:48.229997    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687ae590-f4c4-4aec-97b1-bb60e70299bd" path="/var/lib/kubelet/pods/687ae590-f4c4-4aec-97b1-bb60e70299bd/volumes"
Nov 03 11:31:49.143037 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:49.142999    5393 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 11:31:49.143471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:49.143061    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 11:31:51.730193 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.730166    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:31:51.748547 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.748510    5393 generic.go:356] "Generic (PLEG): container finished" podID="f5d52307-b059-47eb-a13e-c357fb01e355" containerID="a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6" exitCode=0
Nov 03 11:31:51.748771 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.748570    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"
Nov 03 11:31:51.748771 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.748602    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" event={"ID":"f5d52307-b059-47eb-a13e-c357fb01e355","Type":"ContainerDied","Data":"a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6"}
Nov 03 11:31:51.748771 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.748675    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w" event={"ID":"f5d52307-b059-47eb-a13e-c357fb01e355","Type":"ContainerDied","Data":"f51c3cb9963b4744d442ec406bede0bc98ea9e803e7c92dd9a5e566ac7d9cba9"}
Nov 03 11:31:51.748771 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.748710    5393 scope.go:122] "RemoveContainer" containerID="a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6"
Nov 03 11:31:51.761715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.761688    5393 scope.go:122] "RemoveContainer" containerID="a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6"
Nov 03 11:31:51.762150 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:31:51.762079    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6\": container with ID starting with a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6 not found: ID does not exist" containerID="a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6"
Nov 03 11:31:51.762150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.762119    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6"} err="failed to get container status \"a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6\": rpc error: code = NotFound desc = could not find container \"a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6\": container with ID starting with a14e06aa0fbb471999062580e12d73fa1b14ba234b8683a8043fa84fa38386c6 not found: ID does not exist"
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870427    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d52307-b059-47eb-a13e-c357fb01e355-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d52307-b059-47eb-a13e-c357fb01e355-istiod-ca-cert\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870499    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5d52307-b059-47eb-a13e-c357fb01e355-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d52307-b059-47eb-a13e-c357fb01e355-istio-podinfo\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870563    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-data\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870631    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-socket\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870686    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-istio-token\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-istio-token\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870735    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-kube-api-access-hfwck\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-kube-api-access-hfwck\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870764    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-envoy\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870797    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-certs\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.871945 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.870855    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-credential-socket\") pod \"f5d52307-b059-47eb-a13e-c357fb01e355\" (UID: \"f5d52307-b059-47eb-a13e-c357fb01e355\") "
Nov 03 11:31:51.872627 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.872371    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-socket" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.890129 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.872704    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d52307-b059-47eb-a13e-c357fb01e355-istiod-ca-cert" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:31:51.890129 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.873620    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-data" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.890129 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.875515    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-certs" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.890129 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.876555    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-credential-socket" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.934252 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.930126    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-envoy" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:31:51.936257 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.935106    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-istio-token" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:51.941415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.941364    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-kube-api-access-hfwck" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "kube-api-access-hfwck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:31:51.942126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.942082    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5d52307-b059-47eb-a13e-c357fb01e355-istio-podinfo" pod "f5d52307-b059-47eb-a13e-c357fb01e355" (UID: "f5d52307-b059-47eb-a13e-c357fb01e355"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:31:51.957168 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.955440    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"]
Nov 03 11:31:51.972793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972756    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.972793 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972792    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.973068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972806    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.973068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972820    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfwck\" (UniqueName: \"kubernetes.io/projected/f5d52307-b059-47eb-a13e-c357fb01e355-kube-api-access-hfwck\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.973068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972834    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.973068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972845    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.973068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972857    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d52307-b059-47eb-a13e-c357fb01e355-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.973068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972870    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d52307-b059-47eb-a13e-c357fb01e355-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.973068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.972882    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d52307-b059-47eb-a13e-c357fb01e355-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:31:51.974035 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:51.973990    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl" podUID="0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" containerName="payload-processing" containerID="cri-o://4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4" gracePeriod=25
Nov 03 11:31:52.009639 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:52.009591    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh" podUID="10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" containerName="payload-pre-processing" containerID="cri-o://db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17" gracePeriod=25
Nov 03 11:31:52.089995 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:52.086558    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"]
Nov 03 11:31:52.090886 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:52.090855    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ef1662-openshift-default-cf96df6b5-clz5w"]
Nov 03 11:31:52.233248 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:52.233160    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" path="/var/lib/kubelet/pods/f5d52307-b059-47eb-a13e-c357fb01e355/volumes"
Nov 03 11:31:53.627976 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:53.627904    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl" podUID="0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:31:53.628470 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:31:53.628028    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh" podUID="10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:03.628452 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:03.628413    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl" podUID="0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:03.628997 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:03.628490    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh" podUID="10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:06.981053 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:06.980982    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main" containerID="cri-o://cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4" gracePeriod=45
Nov 03 11:32:07.268596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.268570    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 11:32:07.412136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412095    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c3b2439-ed34-4759-a416-95d44a62d63f-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3b2439-ed34-4759-a416-95d44a62d63f-tls-certs\") pod \"9c3b2439-ed34-4759-a416-95d44a62d63f\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") "
Nov 03 11:32:07.412136 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412145    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3b2439-ed34-4759-a416-95d44a62d63f-kube-api-access-dt5xb\" (UniqueName: \"kubernetes.io/projected/9c3b2439-ed34-4759-a416-95d44a62d63f-kube-api-access-dt5xb\") pod \"9c3b2439-ed34-4759-a416-95d44a62d63f\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") "
Nov 03 11:32:07.412419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412197    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-dshm\") pod \"9c3b2439-ed34-4759-a416-95d44a62d63f\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") "
Nov 03 11:32:07.412419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412262    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-model-cache\") pod \"9c3b2439-ed34-4759-a416-95d44a62d63f\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") "
Nov 03 11:32:07.412419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412288    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-home\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-home\") pod \"9c3b2439-ed34-4759-a416-95d44a62d63f\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") "
Nov 03 11:32:07.412419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412314    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-tmp-dir\") pod \"9c3b2439-ed34-4759-a416-95d44a62d63f\" (UID: \"9c3b2439-ed34-4759-a416-95d44a62d63f\") "
Nov 03 11:32:07.415228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412579    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-model-cache" pod "9c3b2439-ed34-4759-a416-95d44a62d63f" (UID: "9c3b2439-ed34-4759-a416-95d44a62d63f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.415228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412595    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-home" pod "9c3b2439-ed34-4759-a416-95d44a62d63f" (UID: "9c3b2439-ed34-4759-a416-95d44a62d63f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.415228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.412778    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-tmp-dir" pod "9c3b2439-ed34-4759-a416-95d44a62d63f" (UID: "9c3b2439-ed34-4759-a416-95d44a62d63f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.415228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.413068    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.415228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.413093    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.415228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.413128    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.418485 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.418434    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3b2439-ed34-4759-a416-95d44a62d63f-tls-certs" pod "9c3b2439-ed34-4759-a416-95d44a62d63f" (UID: "9c3b2439-ed34-4759-a416-95d44a62d63f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:07.419003 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.418973    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3b2439-ed34-4759-a416-95d44a62d63f-kube-api-access-dt5xb" pod "9c3b2439-ed34-4759-a416-95d44a62d63f" (UID: "9c3b2439-ed34-4759-a416-95d44a62d63f"). InnerVolumeSpecName "kube-api-access-dt5xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:07.421141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.421119    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-dshm" pod "9c3b2439-ed34-4759-a416-95d44a62d63f" (UID: "9c3b2439-ed34-4759-a416-95d44a62d63f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:07.514012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.513928    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c3b2439-ed34-4759-a416-95d44a62d63f-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.514012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.513957    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5xb\" (UniqueName: \"kubernetes.io/projected/9c3b2439-ed34-4759-a416-95d44a62d63f-kube-api-access-dt5xb\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.514012 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.513969    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c3b2439-ed34-4759-a416-95d44a62d63f-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:07.856452 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.856420    5393 generic.go:356] "Generic (PLEG): container finished" podID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerID="cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4" exitCode=0
Nov 03 11:32:07.856641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.856471    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"
Nov 03 11:32:07.856641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.856513    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" event={"ID":"9c3b2439-ed34-4759-a416-95d44a62d63f","Type":"ContainerDied","Data":"cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4"}
Nov 03 11:32:07.856641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.856560    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7" event={"ID":"9c3b2439-ed34-4759-a416-95d44a62d63f","Type":"ContainerDied","Data":"645c962aed55823bc829e74889c4867e9dbafd7b457876908a0e500fbd06a0c7"}
Nov 03 11:32:07.856641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.856583    5393 scope.go:122] "RemoveContainer" containerID="cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4"
Nov 03 11:32:07.868179 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.868156    5393 scope.go:122] "RemoveContainer" containerID="cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4"
Nov 03 11:32:07.868480 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:07.868461    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4\": container with ID starting with cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4 not found: ID does not exist" containerID="cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4"
Nov 03 11:32:07.868541 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.868489    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4"} err="failed to get container status \"cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4\": rpc error: code = NotFound desc = could not find container \"cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4\": container with ID starting with cf169227dc83af5b7d0f2978a390649771296be71f89110e8da6c20ab40623e4 not found: ID does not exist"
Nov 03 11:32:07.900369 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.900337    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"]
Nov 03 11:32:07.928412 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:07.928382    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f78621/premium-w0-f78621-kserve-7c9c554d9d-bm6h7"]
Nov 03 11:32:08.230024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:08.229937    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" path="/var/lib/kubelet/pods/9c3b2439-ed34-4759-a416-95d44a62d63f/volumes"
Nov 03 11:32:09.119513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:09.118906    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"]
Nov 03 11:32:12.303448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.303419    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:32:12.370711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.370681    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:32:12.383015 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.382986    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-kube-api-access-bvthc\" (UniqueName: \"kubernetes.io/projected/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-kube-api-access-bvthc\") pod \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\" (UID: \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\") "
Nov 03 11:32:12.383169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.383060    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-kube-api-access-rmrmh\" (UniqueName: \"kubernetes.io/projected/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-kube-api-access-rmrmh\") pod \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\" (UID: \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\") "
Nov 03 11:32:12.383169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.383091    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-plugins-config-volume\") pod \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\" (UID: \"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b\") "
Nov 03 11:32:12.383304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.383196    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-plugins-config-volume\") pod \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\" (UID: \"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc\") "
Nov 03 11:32:12.383559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.383530    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-plugins-config-volume" pod "10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" (UID: "10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:12.383637 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.383530    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-plugins-config-volume" pod "0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" (UID: "0c5cec20-8d50-44f1-b626-b60ce1ca5c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:12.383965 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.383945    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:12.384034 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.383968    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:12.385344 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.385322    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-kube-api-access-bvthc" pod "0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" (UID: "0c5cec20-8d50-44f1-b626-b60ce1ca5c3b"). InnerVolumeSpecName "kube-api-access-bvthc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:12.385575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.385553    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-kube-api-access-rmrmh" pod "10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" (UID: "10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc"). InnerVolumeSpecName "kube-api-access-rmrmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:12.484686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.484649    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvthc\" (UniqueName: \"kubernetes.io/projected/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b-kube-api-access-bvthc\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:12.484686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.484683    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmrmh\" (UniqueName: \"kubernetes.io/projected/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc-kube-api-access-rmrmh\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:12.890813 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.890773    5393 generic.go:356] "Generic (PLEG): container finished" podID="10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" containerID="db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17" exitCode=0
Nov 03 11:32:12.891044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.890876    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh" event={"ID":"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc","Type":"ContainerDied","Data":"db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17"}
Nov 03 11:32:12.891044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.890953    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh" event={"ID":"10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc","Type":"ContainerDied","Data":"92306c1aa5019a4694aaa58e3ce428c5e567d85070a81040278f52971214ff59"}
Nov 03 11:32:12.891044 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.890978    5393 scope.go:122] "RemoveContainer" containerID="db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17"
Nov 03 11:32:12.891303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.891062    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"
Nov 03 11:32:12.893591 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.893551    5393 generic.go:356] "Generic (PLEG): container finished" podID="0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" containerID="4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4" exitCode=0
Nov 03 11:32:12.893721 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.893593    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"
Nov 03 11:32:12.893807 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.893765    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl" event={"ID":"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b","Type":"ContainerDied","Data":"4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4"}
Nov 03 11:32:12.893870 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.893819    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl" event={"ID":"0c5cec20-8d50-44f1-b626-b60ce1ca5c3b","Type":"ContainerDied","Data":"a6f2b30e83b6b82b1366923f88f66bf26f9b2105ade0294f5658f60a66aa4713"}
Nov 03 11:32:12.904055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.903878    5393 scope.go:122] "RemoveContainer" containerID="db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17"
Nov 03 11:32:12.904214 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:12.904186    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17\": container with ID starting with db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17 not found: ID does not exist" containerID="db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17"
Nov 03 11:32:12.904287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.904224    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17"} err="failed to get container status \"db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17\": rpc error: code = NotFound desc = could not find container \"db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17\": container with ID starting with db68c2a15a12801d8846cf8b989ecf41e041f59280557f06666feacab9920f17 not found: ID does not exist"
Nov 03 11:32:12.904287 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.904245    5393 scope.go:122] "RemoveContainer" containerID="4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4"
Nov 03 11:32:12.915033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.915012    5393 scope.go:122] "RemoveContainer" containerID="4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4"
Nov 03 11:32:12.915363 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:12.915340    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4\": container with ID starting with 4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4 not found: ID does not exist" containerID="4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4"
Nov 03 11:32:12.915432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.915371    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4"} err="failed to get container status \"4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4\": rpc error: code = NotFound desc = could not find container \"4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4\": container with ID starting with 4866c56bb0af5c79660bd5850d6d29331a72041642c8acf38e907a7cfb9e6aa4 not found: ID does not exist"
Nov 03 11:32:12.917702 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.917669    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"]
Nov 03 11:32:12.920516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.920491    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fb5766-59557f68d7-6jpvh"]
Nov 03 11:32:12.948946 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.947787    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"]
Nov 03 11:32:12.951790 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:12.951754    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fb5766-fdd496649-vjrxl"]
Nov 03 11:32:13.189310 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:13.189228    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:32:14.231207 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:14.231169    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" path="/var/lib/kubelet/pods/0c5cec20-8d50-44f1-b626-b60ce1ca5c3b/volumes"
Nov 03 11:32:14.231783 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:14.231762    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" path="/var/lib/kubelet/pods/10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc/volumes"
Nov 03 11:32:17.557779 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:17.557744    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"]
Nov 03 11:32:17.558463 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:17.558041    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy" containerID="cri-o://3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82" gracePeriod=30
Nov 03 11:32:19.440081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:19.440032    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"]
Nov 03 11:32:19.449459 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:19.444766    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"]
Nov 03 11:32:21.572220 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:21.572183    5393 scope.go:122] "RemoveContainer" containerID="de47b4bbf4a91b1c47b98936759fd7df96246e9a5295b68971171565d2b08747"
Nov 03 11:32:21.661150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:21.661113    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"]
Nov 03 11:32:21.663381 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:21.661404    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy" containerID="cri-o://2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216" gracePeriod=30
Nov 03 11:32:22.838250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.838189    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 11:32:22.976878 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.976850    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-certs\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.976994    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-kube-api-access-2sjl7\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-kube-api-access-2sjl7\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977022    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4113087-b863-4c49-9942-bfbbc4115fb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4113087-b863-4c49-9942-bfbbc4115fb8-istiod-ca-cert\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977053    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-envoy\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977165    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-certs" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:22.977292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977216    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-credential-socket\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977263    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-data\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977347    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-socket\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977387    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-podinfo\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977417    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-token\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-token\") pod \"c4113087-b863-4c49-9942-bfbbc4115fb8\" (UID: \"c4113087-b863-4c49-9942-bfbbc4115fb8\") "
Nov 03 11:32:22.977638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977500    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-credential-socket" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:22.977638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977535    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4113087-b863-4c49-9942-bfbbc4115fb8-istiod-ca-cert" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:22.977767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.977663    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-socket" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:22.978223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.978185    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4113087-b863-4c49-9942-bfbbc4115fb8-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:22.978223 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.978216    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:22.978404 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.978232    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:22.978404 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.978249    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:22.978404 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.978326    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-data" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:22.979725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.979663    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-envoy" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:22.979725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.979701    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-kube-api-access-2sjl7" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "kube-api-access-2sjl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:22.979859 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.979762    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-token" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:22.980007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.979981    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-podinfo" pod "c4113087-b863-4c49-9942-bfbbc4115fb8" (UID: "c4113087-b863-4c49-9942-bfbbc4115fb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:22.984158 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.984131    5393 generic.go:356] "Generic (PLEG): container finished" podID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerID="3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82" exitCode=0
Nov 03 11:32:22.984271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.984187    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"
Nov 03 11:32:22.984271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.984213    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" event={"ID":"c4113087-b863-4c49-9942-bfbbc4115fb8","Type":"ContainerDied","Data":"3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82"}
Nov 03 11:32:22.984402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.984286    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx" event={"ID":"c4113087-b863-4c49-9942-bfbbc4115fb8","Type":"ContainerDied","Data":"743f78df521c1ce8b2f83646f360e9ca78acb23c660f353ae481d7b90d14a9ea"}
Nov 03 11:32:22.984402 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:22.984318    5393 scope.go:122] "RemoveContainer" containerID="3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82"
Nov 03 11:32:23.004717 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.004689    5393 scope.go:122] "RemoveContainer" containerID="43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c"
Nov 03 11:32:23.018031 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.018000    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"]
Nov 03 11:32:23.018847 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.018816    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6012b5-openshift-default-6c4d9bdd7c-rtpwx"]
Nov 03 11:32:23.021338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.021317    5393 scope.go:122] "RemoveContainer" containerID="3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82"
Nov 03 11:32:23.021694 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:23.021674    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82\": container with ID starting with 3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82 not found: ID does not exist" containerID="3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82"
Nov 03 11:32:23.021809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.021712    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82"} err="failed to get container status \"3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82\": rpc error: code = NotFound desc = could not find container \"3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82\": container with ID starting with 3edbf9637287b026973b5d65ff08276ab00d147beb096928eba56ea2668c2c82 not found: ID does not exist"
Nov 03 11:32:23.021809 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.021751    5393 scope.go:122] "RemoveContainer" containerID="43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c"
Nov 03 11:32:23.022031 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:23.022012    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c\": container with ID starting with 43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c not found: ID does not exist" containerID="43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c"
Nov 03 11:32:23.022080 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.022037    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c"} err="failed to get container status \"43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c\": rpc error: code = NotFound desc = could not find container \"43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c\": container with ID starting with 43230665fd77c4a2986487fa40f668151a29f4869906a5d845c7142e0763279c not found: ID does not exist"
Nov 03 11:32:23.080196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.080168    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjl7\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-kube-api-access-2sjl7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:23.080196 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.080194    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:23.080410 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.080206    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:23.080410 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.080216    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:23.080410 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:23.080224    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4113087-b863-4c49-9942-bfbbc4115fb8-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:24.142909 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.142863    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56" podUID="ee438333-6dc6-40ad-acf8-623a3f1f9164" containerName="main" containerID="cri-o://49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8" gracePeriod=45
Nov 03 11:32:24.229592 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.229555    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" path="/var/lib/kubelet/pods/c4113087-b863-4c49-9942-bfbbc4115fb8/volumes"
Nov 03 11:32:24.423359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.423333    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:32:24.441270 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.441221    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d" podUID="e2d324a2-e483-48ba-a1d5-50a5062b10b2" containerName="payload-processing" containerID="cri-o://664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568" gracePeriod=25
Nov 03 11:32:24.445363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.445329    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q" podUID="f2597418-4682-4239-ad90-371082e13e9b" containerName="payload-pre-processing" containerID="cri-o://738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe" gracePeriod=25
Nov 03 11:32:24.496011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.495980    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee438333-6dc6-40ad-acf8-623a3f1f9164-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee438333-6dc6-40ad-acf8-623a3f1f9164-tls-certs\") pod \"ee438333-6dc6-40ad-acf8-623a3f1f9164\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") "
Nov 03 11:32:24.496195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496026    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-model-cache\") pod \"ee438333-6dc6-40ad-acf8-623a3f1f9164\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") "
Nov 03 11:32:24.496195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496049    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-dshm\") pod \"ee438333-6dc6-40ad-acf8-623a3f1f9164\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") "
Nov 03 11:32:24.496195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496146    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-home\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-home\") pod \"ee438333-6dc6-40ad-acf8-623a3f1f9164\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") "
Nov 03 11:32:24.496362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496205    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee438333-6dc6-40ad-acf8-623a3f1f9164-kube-api-access-tx4bn\" (UniqueName: \"kubernetes.io/projected/ee438333-6dc6-40ad-acf8-623a3f1f9164-kube-api-access-tx4bn\") pod \"ee438333-6dc6-40ad-acf8-623a3f1f9164\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") "
Nov 03 11:32:24.496362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496247    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-tmp-dir\") pod \"ee438333-6dc6-40ad-acf8-623a3f1f9164\" (UID: \"ee438333-6dc6-40ad-acf8-623a3f1f9164\") "
Nov 03 11:32:24.496362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496329    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-model-cache" pod "ee438333-6dc6-40ad-acf8-623a3f1f9164" (UID: "ee438333-6dc6-40ad-acf8-623a3f1f9164"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:24.496540 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496362    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-home" pod "ee438333-6dc6-40ad-acf8-623a3f1f9164" (UID: "ee438333-6dc6-40ad-acf8-623a3f1f9164"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:24.496648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496626    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-tmp-dir" pod "ee438333-6dc6-40ad-acf8-623a3f1f9164" (UID: "ee438333-6dc6-40ad-acf8-623a3f1f9164"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:24.496949 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496894    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:24.497030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496965    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:24.497030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.496981    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:24.498373 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.498340    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee438333-6dc6-40ad-acf8-623a3f1f9164-tls-certs" pod "ee438333-6dc6-40ad-acf8-623a3f1f9164" (UID: "ee438333-6dc6-40ad-acf8-623a3f1f9164"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:24.498477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.498393    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-dshm" pod "ee438333-6dc6-40ad-acf8-623a3f1f9164" (UID: "ee438333-6dc6-40ad-acf8-623a3f1f9164"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:24.498477 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.498457    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee438333-6dc6-40ad-acf8-623a3f1f9164-kube-api-access-tx4bn" pod "ee438333-6dc6-40ad-acf8-623a3f1f9164" (UID: "ee438333-6dc6-40ad-acf8-623a3f1f9164"). InnerVolumeSpecName "kube-api-access-tx4bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:24.597684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.597647    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee438333-6dc6-40ad-acf8-623a3f1f9164-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:24.597684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.597679    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee438333-6dc6-40ad-acf8-623a3f1f9164-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:24.597684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.597690    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx4bn\" (UniqueName: \"kubernetes.io/projected/ee438333-6dc6-40ad-acf8-623a3f1f9164-kube-api-access-tx4bn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:24.998355 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.998321    5393 generic.go:356] "Generic (PLEG): container finished" podID="ee438333-6dc6-40ad-acf8-623a3f1f9164" containerID="49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8" exitCode=0
Nov 03 11:32:24.998548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.998413    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"
Nov 03 11:32:24.998548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.998407    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56" event={"ID":"ee438333-6dc6-40ad-acf8-623a3f1f9164","Type":"ContainerDied","Data":"49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8"}
Nov 03 11:32:24.998548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.998532    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56" event={"ID":"ee438333-6dc6-40ad-acf8-623a3f1f9164","Type":"ContainerDied","Data":"b560ac0926847d0f867a2cdfd5e66f1d09adc654e6bcb0577df6f02cbdd16e13"}
Nov 03 11:32:24.998681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:24.998551    5393 scope.go:122] "RemoveContainer" containerID="49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8"
Nov 03 11:32:25.010173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:25.010151    5393 scope.go:122] "RemoveContainer" containerID="49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8"
Nov 03 11:32:25.010494 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:25.010472    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8\": container with ID starting with 49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8 not found: ID does not exist" containerID="49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8"
Nov 03 11:32:25.010561 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:25.010503    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8"} err="failed to get container status \"49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8\": rpc error: code = NotFound desc = could not find container \"49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8\": container with ID starting with 49dce3d0fb97d8adf3f3ff6ff30f64ce13df0338096d5019ba703b7907ef71a8 not found: ID does not exist"
Nov 03 11:32:25.020221 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:25.020194    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"]
Nov 03 11:32:25.025128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:25.025102    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-836589/sub-test-model-836589-kserve-5d6565cfc5-bns56"]
Nov 03 11:32:26.238736 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:26.238701    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee438333-6dc6-40ad-acf8-623a3f1f9164" path="/var/lib/kubelet/pods/ee438333-6dc6-40ad-acf8-623a3f1f9164/volumes"
Nov 03 11:32:26.945613 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:26.945557    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 11:32:27.014295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.014260    5393 generic.go:356] "Generic (PLEG): container finished" podID="6469e418-320e-4d01-86a4-99b169b7fc34" containerID="2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216" exitCode=0
Nov 03 11:32:27.014471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.014318    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"
Nov 03 11:32:27.014471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.014356    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" event={"ID":"6469e418-320e-4d01-86a4-99b169b7fc34","Type":"ContainerDied","Data":"2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216"}
Nov 03 11:32:27.014471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.014425    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7" event={"ID":"6469e418-320e-4d01-86a4-99b169b7fc34","Type":"ContainerDied","Data":"c6e2d9d0f3456ec8370294c90129d74329e213ca415d9d1fc1b0b610a349379c"}
Nov 03 11:32:27.014471 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.014439    5393 scope.go:122] "RemoveContainer" containerID="2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216"
Nov 03 11:32:27.024173 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024149    5393 scope.go:122] "RemoveContainer" containerID="ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296"
Nov 03 11:32:27.024536 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024517    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-data\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.024633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024600    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6469e418-320e-4d01-86a4-99b169b7fc34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6469e418-320e-4d01-86a4-99b169b7fc34-istio-podinfo\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.024697 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024648    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-istio-token\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-istio-token\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.024697 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024680    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-kube-api-access-bnm9z\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-kube-api-access-bnm9z\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.024806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024785    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-data" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.024861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024802    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-certs\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.024861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024833    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-socket\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.025014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024898    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6469e418-320e-4d01-86a4-99b169b7fc34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6469e418-320e-4d01-86a4-99b169b7fc34-istiod-ca-cert\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.025014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.024958    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-credential-socket\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.025126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.025036    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-envoy\") pod \"6469e418-320e-4d01-86a4-99b169b7fc34\" (UID: \"6469e418-320e-4d01-86a4-99b169b7fc34\") "
Nov 03 11:32:27.025181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.025141    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-certs" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.025601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.025466    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-credential-socket" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.025601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.025546    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.025601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.025556    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-socket" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.025601 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.025568    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.026114 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.025659    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6469e418-320e-4d01-86a4-99b169b7fc34-istiod-ca-cert" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:27.027236 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.027206    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6469e418-320e-4d01-86a4-99b169b7fc34-istio-podinfo" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:27.027733 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.027707    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-istio-token" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:27.027849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.027763    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-envoy" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:27.028191 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.028174    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-kube-api-access-bnm9z" pod "6469e418-320e-4d01-86a4-99b169b7fc34" (UID: "6469e418-320e-4d01-86a4-99b169b7fc34"). InnerVolumeSpecName "kube-api-access-bnm9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:27.070292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.070256    5393 scope.go:122] "RemoveContainer" containerID="2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216"
Nov 03 11:32:27.070724 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:27.070700    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216\": container with ID starting with 2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216 not found: ID does not exist" containerID="2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216"
Nov 03 11:32:27.070822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.070741    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216"} err="failed to get container status \"2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216\": rpc error: code = NotFound desc = could not find container \"2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216\": container with ID starting with 2464afc74b0b51e7915262dc624ed5b93d9213d90810c40a5f14ae28596f1216 not found: ID does not exist"
Nov 03 11:32:27.070822 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.070767    5393 scope.go:122] "RemoveContainer" containerID="ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296"
Nov 03 11:32:27.072042 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:27.071996    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296\": container with ID starting with ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296 not found: ID does not exist" containerID="ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296"
Nov 03 11:32:27.072179 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.072039    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296"} err="failed to get container status \"ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296\": rpc error: code = NotFound desc = could not find container \"ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296\": container with ID starting with ff030db48a509745e3d347ffa6614f2ba0c388e7f698baafd85e13430ca36296 not found: ID does not exist"
Nov 03 11:32:27.127225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.127188    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6469e418-320e-4d01-86a4-99b169b7fc34-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.127225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.127224    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.127436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.127237    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.127436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.127249    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6469e418-320e-4d01-86a4-99b169b7fc34-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.127436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.127262    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.127436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.127275    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnm9z\" (UniqueName: \"kubernetes.io/projected/6469e418-320e-4d01-86a4-99b169b7fc34-kube-api-access-bnm9z\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.127436 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.127288    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6469e418-320e-4d01-86a4-99b169b7fc34-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:27.338249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.338210    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"]
Nov 03 11:32:27.340573 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:27.340549    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f78621-openshift-default-74d8d5d569-jkmg7"]
Nov 03 11:32:28.183064 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:28.183029    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"]
Nov 03 11:32:28.249180 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:28.249143    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" path="/var/lib/kubelet/pods/6469e418-320e-4d01-86a4-99b169b7fc34/volumes"
Nov 03 11:32:28.603558 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:28.603522    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"]
Nov 03 11:32:28.604110 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:28.603764    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy" containerID="cri-o://b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d" gracePeriod=30
Nov 03 11:32:33.183675 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.183631    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m" podUID="941c0d96-a831-41d5-9392-7802117b42ad" containerName="payload-pre-processing" containerID="cri-o://bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1" gracePeriod=25
Nov 03 11:32:33.428578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.428540    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d" podUID="e2d324a2-e483-48ba-a1d5-50a5062b10b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:33.886792 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.886723    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:32:33.919818 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.919785    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-podinfo\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.920024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.919841    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-envoy\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.920024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.919883    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-token\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-token\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.920024 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.919932    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9fafe7-e074-44a0-953f-a53b2b47629a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9fafe7-e074-44a0-953f-a53b2b47629a-istiod-ca-cert\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.920318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.920040    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-data\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.920318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.920075    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-certs\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.920318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.920128    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-socket\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.920318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.920156    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-kube-api-access-brnmt\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-kube-api-access-brnmt\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.920318 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.920198    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-credential-socket\") pod \"1e9fafe7-e074-44a0-953f-a53b2b47629a\" (UID: \"1e9fafe7-e074-44a0-953f-a53b2b47629a\") "
Nov 03 11:32:33.921142 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.921011    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-socket" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:33.921315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.921286    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-certs" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:33.921415 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.921395    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9fafe7-e074-44a0-953f-a53b2b47629a-istiod-ca-cert" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:33.921575 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.921557    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-credential-socket" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:33.923449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.922744    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-data" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:33.923449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.923133    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-podinfo" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:33.923449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.923232    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-envoy" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:33.923449 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.923427    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-token" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:33.923700 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:33.923675    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-kube-api-access-brnmt" pod "1e9fafe7-e074-44a0-953f-a53b2b47629a" (UID: "1e9fafe7-e074-44a0-953f-a53b2b47629a"). InnerVolumeSpecName "kube-api-access-brnmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:34.021523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021488    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brnmt\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-kube-api-access-brnmt\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.021523 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021522    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.021768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021534    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.021768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021543    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.021768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021554    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.021768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021562    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9fafe7-e074-44a0-953f-a53b2b47629a-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.021768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021570    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.021768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021579    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.021768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.021593    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9fafe7-e074-44a0-953f-a53b2b47629a-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:34.062204 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.062172    5393 generic.go:356] "Generic (PLEG): container finished" podID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerID="b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d" exitCode=0
Nov 03 11:32:34.062394 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.062215    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" event={"ID":"1e9fafe7-e074-44a0-953f-a53b2b47629a","Type":"ContainerDied","Data":"b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d"}
Nov 03 11:32:34.062394 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.062268    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn" event={"ID":"1e9fafe7-e074-44a0-953f-a53b2b47629a","Type":"ContainerDied","Data":"6b6c6a784672da91aa3bce3f475853203d7f4641811de5cbd1c3b46b3475ca9e"}
Nov 03 11:32:34.062394 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.062283    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"
Nov 03 11:32:34.062394 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.062298    5393 scope.go:122] "RemoveContainer" containerID="b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d"
Nov 03 11:32:34.074377 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.074351    5393 scope.go:122] "RemoveContainer" containerID="79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b"
Nov 03 11:32:34.084904 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.084880    5393 scope.go:122] "RemoveContainer" containerID="b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d"
Nov 03 11:32:34.085233 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:34.085206    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d\": container with ID starting with b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d not found: ID does not exist" containerID="b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d"
Nov 03 11:32:34.085304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.085242    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d"} err="failed to get container status \"b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d\": rpc error: code = NotFound desc = could not find container \"b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d\": container with ID starting with b0717c2dbd87de4659729841c29903648fbe2a9475a409dee5e1626e68abb51d not found: ID does not exist"
Nov 03 11:32:34.085304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.085264    5393 scope.go:122] "RemoveContainer" containerID="79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b"
Nov 03 11:32:34.085546 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:34.085526    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b\": container with ID starting with 79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b not found: ID does not exist" containerID="79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b"
Nov 03 11:32:34.085614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.085556    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b"} err="failed to get container status \"79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b\": rpc error: code = NotFound desc = could not find container \"79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b\": container with ID starting with 79d3fb9f690869ce1a36edc462e6ab75a069175f8005887975578c083baad12b not found: ID does not exist"
Nov 03 11:32:34.086052 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.086028    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"]
Nov 03 11:32:34.089935 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.089893    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-35936b-openshift-default-5f7dd5fc45-n66nn"]
Nov 03 11:32:34.230164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.230082    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" path="/var/lib/kubelet/pods/1e9fafe7-e074-44a0-953f-a53b2b47629a/volumes"
Nov 03 11:32:34.443720 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:34.443682    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q" podUID="f2597418-4682-4239-ad90-371082e13e9b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:39.260324 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:39.260290    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"]
Nov 03 11:32:39.272849 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:39.272817    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"]
Nov 03 11:32:40.993103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:40.993063    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m" podUID="941c0d96-a831-41d5-9392-7802117b42ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:43.198486 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:43.198447    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:32:43.428719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:43.428684    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d" podUID="e2d324a2-e483-48ba-a1d5-50a5062b10b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:44.261473 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.261421    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh" podUID="9d5239bc-ba6c-4d26-b548-232d9252f273" containerName="payload-processing" containerID="cri-o://69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e" gracePeriod=25
Nov 03 11:32:44.273989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.273951    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn" podUID="25c86b89-8fd8-499a-a6a5-6846dd17ac26" containerName="payload-pre-processing" containerID="cri-o://7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145" gracePeriod=25
Nov 03 11:32:44.444734 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.444689    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q" podUID="f2597418-4682-4239-ad90-371082e13e9b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:44.486831 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:44.486797    5393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2597418_4682_4239_ad90_371082e13e9b.slice/crio-738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d324a2_e483_48ba_a1d5_50a5062b10b2.slice/crio-664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 11:32:44.639944 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.639900    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:32:44.688661 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.688626    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:32:44.775493 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.775448    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d324a2-e483-48ba-a1d5-50a5062b10b2-kube-api-access-7s4rr\" (UniqueName: \"kubernetes.io/projected/e2d324a2-e483-48ba-a1d5-50a5062b10b2-kube-api-access-7s4rr\") pod \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\" (UID: \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\") "
Nov 03 11:32:44.775715 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.775521    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d324a2-e483-48ba-a1d5-50a5062b10b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d324a2-e483-48ba-a1d5-50a5062b10b2-plugins-config-volume\") pod \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\" (UID: \"e2d324a2-e483-48ba-a1d5-50a5062b10b2\") "
Nov 03 11:32:44.775943 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.775896    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d324a2-e483-48ba-a1d5-50a5062b10b2-plugins-config-volume" pod "e2d324a2-e483-48ba-a1d5-50a5062b10b2" (UID: "e2d324a2-e483-48ba-a1d5-50a5062b10b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:44.776086 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.776066    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d324a2-e483-48ba-a1d5-50a5062b10b2-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:44.778176 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.778143    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d324a2-e483-48ba-a1d5-50a5062b10b2-kube-api-access-7s4rr" pod "e2d324a2-e483-48ba-a1d5-50a5062b10b2" (UID: "e2d324a2-e483-48ba-a1d5-50a5062b10b2"). InnerVolumeSpecName "kube-api-access-7s4rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:44.877345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.877309    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2597418-4682-4239-ad90-371082e13e9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2597418-4682-4239-ad90-371082e13e9b-plugins-config-volume\") pod \"f2597418-4682-4239-ad90-371082e13e9b\" (UID: \"f2597418-4682-4239-ad90-371082e13e9b\") "
Nov 03 11:32:44.877549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.877367    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2597418-4682-4239-ad90-371082e13e9b-kube-api-access-6f882\" (UniqueName: \"kubernetes.io/projected/f2597418-4682-4239-ad90-371082e13e9b-kube-api-access-6f882\") pod \"f2597418-4682-4239-ad90-371082e13e9b\" (UID: \"f2597418-4682-4239-ad90-371082e13e9b\") "
Nov 03 11:32:44.877837 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.877807    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2597418-4682-4239-ad90-371082e13e9b-plugins-config-volume" pod "f2597418-4682-4239-ad90-371082e13e9b" (UID: "f2597418-4682-4239-ad90-371082e13e9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:44.877937 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.877823    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4rr\" (UniqueName: \"kubernetes.io/projected/e2d324a2-e483-48ba-a1d5-50a5062b10b2-kube-api-access-7s4rr\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:44.879804 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.879781    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2597418-4682-4239-ad90-371082e13e9b-kube-api-access-6f882" pod "f2597418-4682-4239-ad90-371082e13e9b" (UID: "f2597418-4682-4239-ad90-371082e13e9b"). InnerVolumeSpecName "kube-api-access-6f882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:44.978787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.978746    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2597418-4682-4239-ad90-371082e13e9b-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:44.978787 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:44.978779    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f882\" (UniqueName: \"kubernetes.io/projected/f2597418-4682-4239-ad90-371082e13e9b-kube-api-access-6f882\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:45.170658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.170566    5393 generic.go:356] "Generic (PLEG): container finished" podID="f2597418-4682-4239-ad90-371082e13e9b" containerID="738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe" exitCode=0
Nov 03 11:32:45.170658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.170612    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"
Nov 03 11:32:45.170884 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.170650    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q" event={"ID":"f2597418-4682-4239-ad90-371082e13e9b","Type":"ContainerDied","Data":"738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe"}
Nov 03 11:32:45.170884 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.170740    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q" event={"ID":"f2597418-4682-4239-ad90-371082e13e9b","Type":"ContainerDied","Data":"5b39b9d4d6ea4e5ed0164e24770342feace51f7a22f85869af45dcd22d7604ae"}
Nov 03 11:32:45.170884 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.170766    5393 scope.go:122] "RemoveContainer" containerID="738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe"
Nov 03 11:32:45.172299 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.172275    5393 generic.go:356] "Generic (PLEG): container finished" podID="e2d324a2-e483-48ba-a1d5-50a5062b10b2" containerID="664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568" exitCode=0
Nov 03 11:32:45.172420 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.172344    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"
Nov 03 11:32:45.172420 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.172366    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d" event={"ID":"e2d324a2-e483-48ba-a1d5-50a5062b10b2","Type":"ContainerDied","Data":"664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568"}
Nov 03 11:32:45.172548 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.172420    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d" event={"ID":"e2d324a2-e483-48ba-a1d5-50a5062b10b2","Type":"ContainerDied","Data":"ad5472c847c44b6250c6afde5120723e5e07b1ada994124879c182535a7c9599"}
Nov 03 11:32:45.182382 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.182362    5393 scope.go:122] "RemoveContainer" containerID="738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe"
Nov 03 11:32:45.182734 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:45.182707    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe\": container with ID starting with 738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe not found: ID does not exist" containerID="738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe"
Nov 03 11:32:45.182830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.182747    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe"} err="failed to get container status \"738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe\": rpc error: code = NotFound desc = could not find container \"738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe\": container with ID starting with 738da4ef3ace546a978bf5ff24898bd49235033069b92eff65bf33fe9264acbe not found: ID does not exist"
Nov 03 11:32:45.182830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.182768    5393 scope.go:122] "RemoveContainer" containerID="664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568"
Nov 03 11:32:45.192065 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.192043    5393 scope.go:122] "RemoveContainer" containerID="664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568"
Nov 03 11:32:45.192342 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:45.192320    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568\": container with ID starting with 664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568 not found: ID does not exist" containerID="664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568"
Nov 03 11:32:45.192410 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.192354    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568"} err="failed to get container status \"664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568\": rpc error: code = NotFound desc = could not find container \"664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568\": container with ID starting with 664f0069a84d4d4183a66e60e1da3b5ed70827f6cb323cf98256e79c91822568 not found: ID does not exist"
Nov 03 11:32:45.200078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.200058    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"]
Nov 03 11:32:45.205425 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.205400    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-35936b-6c4b8b9b95-fdb4q"]
Nov 03 11:32:45.215357 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.215338    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"]
Nov 03 11:32:45.220908 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:45.220887    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-35936b-555d5645bc-6nc7d"]
Nov 03 11:32:46.230404 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:46.230371    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d324a2-e483-48ba-a1d5-50a5062b10b2" path="/var/lib/kubelet/pods/e2d324a2-e483-48ba-a1d5-50a5062b10b2/volumes"
Nov 03 11:32:46.230835 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:46.230821    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2597418-4682-4239-ad90-371082e13e9b" path="/var/lib/kubelet/pods/f2597418-4682-4239-ad90-371082e13e9b/volumes"
Nov 03 11:32:47.768764 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:47.768730    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh" podUID="9d5239bc-ba6c-4d26-b548-232d9252f273" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:47.769247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:47.768741    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn" podUID="25c86b89-8fd8-499a-a6a5-6846dd17ac26" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:49.320259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:49.320216    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"]
Nov 03 11:32:49.321056 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:49.320477    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy" containerID="cri-o://83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab" gracePeriod=30
Nov 03 11:32:50.422017 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.421972    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"]
Nov 03 11:32:50.422574 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.422233    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy" containerID="cri-o://96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c" gracePeriod=30
Nov 03 11:32:50.931898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.931848    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"]
Nov 03 11:32:50.932440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932422    5393 cpu_manager.go:414] "containerMap: removing container" podUID="10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" containerName="payload-pre-processing"
Nov 03 11:32:50.932440 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932443    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" containerName="payload-pre-processing"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932459    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e2d324a2-e483-48ba-a1d5-50a5062b10b2" containerName="payload-processing"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932465    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d324a2-e483-48ba-a1d5-50a5062b10b2" containerName="payload-processing"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932474    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" containerName="payload-processing"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932480    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" containerName="payload-processing"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932487    5393 cpu_manager.go:414] "containerMap: removing container" podUID="687ae590-f4c4-4aec-97b1-bb60e70299bd" containerName="revoke-keys"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932492    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687ae590-f4c4-4aec-97b1-bb60e70299bd" containerName="revoke-keys"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932499    5393 cpu_manager.go:414] "containerMap: removing container" podUID="b2aa0412-2a7f-4850-ba77-92f35382670f" containerName="payload-processing"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932504    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2aa0412-2a7f-4850-ba77-92f35382670f" containerName="payload-processing"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932511    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932515    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932521    5393 cpu_manager.go:414] "containerMap: removing container" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932526    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932539    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932546    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932563    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy"
Nov 03 11:32:50.932570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932570    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932580    5393 cpu_manager.go:414] "containerMap: removing container" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932585    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932594    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f2597418-4682-4239-ad90-371082e13e9b" containerName="payload-pre-processing"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932598    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2597418-4682-4239-ad90-371082e13e9b" containerName="payload-pre-processing"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932608    5393 cpu_manager.go:414] "containerMap: removing container" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932612    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932623    5393 cpu_manager.go:414] "containerMap: removing container" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932628    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932635    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ee438333-6dc6-40ad-acf8-623a3f1f9164" containerName="main"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932639    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee438333-6dc6-40ad-acf8-623a3f1f9164" containerName="main"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932650    5393 cpu_manager.go:414] "containerMap: removing container" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932655    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932727    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d324a2-e483-48ba-a1d5-50a5062b10b2" containerName="payload-processing"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932735    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5cec20-8d50-44f1-b626-b60ce1ca5c3b" containerName="payload-processing"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932744    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="687ae590-f4c4-4aec-97b1-bb60e70299bd" containerName="revoke-keys"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932750    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2aa0412-2a7f-4850-ba77-92f35382670f" containerName="payload-processing"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932755    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3b2439-ed34-4759-a416-95d44a62d63f" containerName="main"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932761    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932767    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932773    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="432522bc-6cbe-4cb5-a30c-9f9fad1b6be5" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932780    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932785    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4113087-b863-4c49-9942-bfbbc4115fb8" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932790    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2597418-4682-4239-ad90-371082e13e9b" containerName="payload-pre-processing"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932796    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d52307-b059-47eb-a13e-c357fb01e355" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932802    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee438333-6dc6-40ad-acf8-623a3f1f9164" containerName="main"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932808    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9fafe7-e074-44a0-953f-a53b2b47629a" containerName="istio-proxy"
Nov 03 11:32:50.933088 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.932813    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b2ea3b-ee96-4d6d-9a9b-8721d5e6c0bc" containerName="payload-pre-processing"
Nov 03 11:32:50.935891 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.935873    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:50.944815 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.944771    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:50.944994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.944891    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:50.944994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.944943    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpmn\" (UniqueName: \"kubernetes.io/projected/2812357e-7c18-4d0f-afd6-b419cdefd21f-kube-api-access-gvpmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:50.945767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.945744    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:32:50.945878 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.945856    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:32:50.946292 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.946269    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"]
Nov 03 11:32:50.993213 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:50.993175    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m" podUID="941c0d96-a831-41d5-9392-7802117b42ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:51.059345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:51.055579    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:51.059345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:51.055738    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:51.059345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:51.055785    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpmn\" (UniqueName: \"kubernetes.io/projected/2812357e-7c18-4d0f-afd6-b419cdefd21f-kube-api-access-gvpmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:51.059345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:51.058385    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:51.064340 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:51.064208    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:51.066761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:51.066730    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpmn\" (UniqueName: \"kubernetes.io/projected/2812357e-7c18-4d0f-afd6-b419cdefd21f-kube-api-access-gvpmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:51.253081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:51.252997    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:51.527660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:51.527635    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"]
Nov 03 11:32:51.529408 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:32:51.529381    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2812357e_7c18_4d0f_afd6_b419cdefd21f.slice/crio-a6107928ef007ef90f4b3c7e91656f805bff16f12fa072492a00933e2dcd7b2f WatchSource:0}: Error finding container a6107928ef007ef90f4b3c7e91656f805bff16f12fa072492a00933e2dcd7b2f: Status 404 returned error can't find the container with id a6107928ef007ef90f4b3c7e91656f805bff16f12fa072492a00933e2dcd7b2f
Nov 03 11:32:52.221265 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:52.221228    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll" event={"ID":"2812357e-7c18-4d0f-afd6-b419cdefd21f","Type":"ContainerStarted","Data":"6d5433c4645be6feaefc252e1f873f8fe9dd6450c1c7915b32fde8a66ceaff8f"}
Nov 03 11:32:52.221489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:52.221277    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll" event={"ID":"2812357e-7c18-4d0f-afd6-b419cdefd21f","Type":"ContainerStarted","Data":"a6107928ef007ef90f4b3c7e91656f805bff16f12fa072492a00933e2dcd7b2f"}
Nov 03 11:32:52.242167 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:52.242122    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll" podStartSLOduration=2.242109182 podStartE2EDuration="2.242109182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:32:52.238699557 +0000 UTC m=+3816.143553999" watchObservedRunningTime="2026-11-03 11:32:52.242109182 +0000 UTC m=+3816.146963622"
Nov 03 11:32:53.228879 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.228842    5393 generic.go:356] "Generic (PLEG): container finished" podID="2812357e-7c18-4d0f-afd6-b419cdefd21f" containerID="6d5433c4645be6feaefc252e1f873f8fe9dd6450c1c7915b32fde8a66ceaff8f" exitCode=0
Nov 03 11:32:53.229375 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.228939    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll" event={"ID":"2812357e-7c18-4d0f-afd6-b419cdefd21f","Type":"ContainerDied","Data":"6d5433c4645be6feaefc252e1f873f8fe9dd6450c1c7915b32fde8a66ceaff8f"}
Nov 03 11:32:53.452033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.451978    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 11:32:53.481882 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.481840    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941c0d96-a831-41d5-9392-7802117b42ad-kube-api-access-tvcr7\" (UniqueName: \"kubernetes.io/projected/941c0d96-a831-41d5-9392-7802117b42ad-kube-api-access-tvcr7\") pod \"941c0d96-a831-41d5-9392-7802117b42ad\" (UID: \"941c0d96-a831-41d5-9392-7802117b42ad\") "
Nov 03 11:32:53.482076 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.481968    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941c0d96-a831-41d5-9392-7802117b42ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941c0d96-a831-41d5-9392-7802117b42ad-plugins-config-volume\") pod \"941c0d96-a831-41d5-9392-7802117b42ad\" (UID: \"941c0d96-a831-41d5-9392-7802117b42ad\") "
Nov 03 11:32:53.482328 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.482306    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941c0d96-a831-41d5-9392-7802117b42ad-plugins-config-volume" pod "941c0d96-a831-41d5-9392-7802117b42ad" (UID: "941c0d96-a831-41d5-9392-7802117b42ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:53.482442 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.482428    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/941c0d96-a831-41d5-9392-7802117b42ad-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:53.484466 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.484428    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941c0d96-a831-41d5-9392-7802117b42ad-kube-api-access-tvcr7" pod "941c0d96-a831-41d5-9392-7802117b42ad" (UID: "941c0d96-a831-41d5-9392-7802117b42ad"). InnerVolumeSpecName "kube-api-access-tvcr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:53.583186 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:53.583149    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcr7\" (UniqueName: \"kubernetes.io/projected/941c0d96-a831-41d5-9392-7802117b42ad-kube-api-access-tvcr7\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.236656 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.236617    5393 generic.go:356] "Generic (PLEG): container finished" podID="941c0d96-a831-41d5-9392-7802117b42ad" containerID="bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1" exitCode=0
Nov 03 11:32:54.237181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.236675    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"
Nov 03 11:32:54.237181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.236693    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m" event={"ID":"941c0d96-a831-41d5-9392-7802117b42ad","Type":"ContainerDied","Data":"bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1"}
Nov 03 11:32:54.237181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.236743    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m" event={"ID":"941c0d96-a831-41d5-9392-7802117b42ad","Type":"ContainerDied","Data":"543062c540c944ee7a36bebd3505fe6575f4fdeab80424252247f65f7ef813a9"}
Nov 03 11:32:54.237181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.236763    5393 scope.go:122] "RemoveContainer" containerID="bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1"
Nov 03 11:32:54.247989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.247499    5393 scope.go:122] "RemoveContainer" containerID="bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1"
Nov 03 11:32:54.247989 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:54.247852    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1\": container with ID starting with bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1 not found: ID does not exist" containerID="bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1"
Nov 03 11:32:54.247989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.247896    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1"} err="failed to get container status \"bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1\": rpc error: code = NotFound desc = could not find container \"bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1\": container with ID starting with bd5670ac99215f14a78a14fa5e65a4d5a743328e63dd1129219bc2a606b47ff1 not found: ID does not exist"
Nov 03 11:32:54.262874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.262834    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"]
Nov 03 11:32:54.265527 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.265501    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cc8214-7dd6fcf467-t9w7m"]
Nov 03 11:32:54.443840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.443820    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:54.493285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.493180    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-xks-ca\") pod \"2812357e-7c18-4d0f-afd6-b419cdefd21f\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") "
Nov 03 11:32:54.493285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.493222    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-openshift-ca\") pod \"2812357e-7c18-4d0f-afd6-b419cdefd21f\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") "
Nov 03 11:32:54.493549 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.493341    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2812357e-7c18-4d0f-afd6-b419cdefd21f-kube-api-access-gvpmn\" (UniqueName: \"kubernetes.io/projected/2812357e-7c18-4d0f-afd6-b419cdefd21f-kube-api-access-gvpmn\") pod \"2812357e-7c18-4d0f-afd6-b419cdefd21f\" (UID: \"2812357e-7c18-4d0f-afd6-b419cdefd21f\") "
Nov 03 11:32:54.494147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.494114    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-openshift-ca" pod "2812357e-7c18-4d0f-afd6-b419cdefd21f" (UID: "2812357e-7c18-4d0f-afd6-b419cdefd21f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:54.495993 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.495966    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-xks-ca" pod "2812357e-7c18-4d0f-afd6-b419cdefd21f" (UID: "2812357e-7c18-4d0f-afd6-b419cdefd21f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:32:54.496277 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.496250    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2812357e-7c18-4d0f-afd6-b419cdefd21f-kube-api-access-gvpmn" pod "2812357e-7c18-4d0f-afd6-b419cdefd21f" (UID: "2812357e-7c18-4d0f-afd6-b419cdefd21f"). InnerVolumeSpecName "kube-api-access-gvpmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:54.574437 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.574411    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 11:32:54.594876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.594844    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpmn\" (UniqueName: \"kubernetes.io/projected/2812357e-7c18-4d0f-afd6-b419cdefd21f-kube-api-access-gvpmn\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.594876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.594878    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.595172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.594892    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2812357e-7c18-4d0f-afd6-b419cdefd21f-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.695732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.695696    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-data\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.695732 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.695736    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-certs\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.696033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.695777    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-kube-api-access-vpj9j\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-kube-api-access-vpj9j\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.696033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.695842    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-socket\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.696033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.695874    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-credential-socket\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.696033 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.695995    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-token\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-token\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.696271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696045    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-data" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:54.696271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696069    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istiod-ca-cert\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.696271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696079    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-certs" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:54.696271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696131    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-envoy\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.696271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696216    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-podinfo\") pod \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\" (UID: \"4c7724d8-aa88-49c8-9a08-844b43f5a00d\") "
Nov 03 11:32:54.696271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696221    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-credential-socket" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:54.696271 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696238    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-socket" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:54.696676 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696468    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istiod-ca-cert" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:54.696760 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696733    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.697005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696760    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.697005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696775    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.697005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696790    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.697005 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.696822    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.698615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.698590    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-envoy" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:54.698714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.698610    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-token" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:54.698714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.698645    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-kube-api-access-vpj9j" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "kube-api-access-vpj9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:54.698714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.698658    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-podinfo" pod "4c7724d8-aa88-49c8-9a08-844b43f5a00d" (UID: "4c7724d8-aa88-49c8-9a08-844b43f5a00d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:54.797838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.797751    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpj9j\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-kube-api-access-vpj9j\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.797838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.797782    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.797838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.797791    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.797838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.797803    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7724d8-aa88-49c8-9a08-844b43f5a00d-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:54.913994 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.913955    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"]
Nov 03 11:32:54.922648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.922615    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"]
Nov 03 11:32:54.940767 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:54.940734    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"]
Nov 03 11:32:55.246733 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.246698    5393 generic.go:356] "Generic (PLEG): container finished" podID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerID="83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab" exitCode=0
Nov 03 11:32:55.247333 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.246787    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"
Nov 03 11:32:55.247333 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.246787    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" event={"ID":"4c7724d8-aa88-49c8-9a08-844b43f5a00d","Type":"ContainerDied","Data":"83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab"}
Nov 03 11:32:55.247333 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.246856    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps" event={"ID":"4c7724d8-aa88-49c8-9a08-844b43f5a00d","Type":"ContainerDied","Data":"82bbd8940470540c66017bcbe5030b5f7bd50cab16a524f70a61ed8fa8aa0544"}
Nov 03 11:32:55.247333 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.246890    5393 scope.go:122] "RemoveContainer" containerID="83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab"
Nov 03 11:32:55.248467 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.248435    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll" event={"ID":"2812357e-7c18-4d0f-afd6-b419cdefd21f","Type":"ContainerDied","Data":"a6107928ef007ef90f4b3c7e91656f805bff16f12fa072492a00933e2dcd7b2f"}
Nov 03 11:32:55.248615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.248492    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6107928ef007ef90f4b3c7e91656f805bff16f12fa072492a00933e2dcd7b2f"
Nov 03 11:32:55.248615 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.248551    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"
Nov 03 11:32:55.258166 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.258065    5393 scope.go:122] "RemoveContainer" containerID="c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4"
Nov 03 11:32:55.268651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.268622    5393 scope.go:122] "RemoveContainer" containerID="83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab"
Nov 03 11:32:55.269022 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:55.268998    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab\": container with ID starting with 83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab not found: ID does not exist" containerID="83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab"
Nov 03 11:32:55.269126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.269036    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab"} err="failed to get container status \"83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab\": rpc error: code = NotFound desc = could not find container \"83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab\": container with ID starting with 83577c63959aece23f818f7fc01a76f17c80dd2ae727ae9f230aa379533337ab not found: ID does not exist"
Nov 03 11:32:55.269126 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.269063    5393 scope.go:122] "RemoveContainer" containerID="c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4"
Nov 03 11:32:55.269385 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:55.269363    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4\": container with ID starting with c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4 not found: ID does not exist" containerID="c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4"
Nov 03 11:32:55.269442 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.269390    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4"} err="failed to get container status \"c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4\": rpc error: code = NotFound desc = could not find container \"c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4\": container with ID starting with c4fd8780bf803b9bdb636f50fe8df39c17371ce443371ece186a2935ce20ffa4 not found: ID does not exist"
Nov 03 11:32:55.279742 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.279714    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"]
Nov 03 11:32:55.285756 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.285721    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cc8214-openshift-default-6b7ff6457c-gj5ps"]
Nov 03 11:32:55.716850 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.716825    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:32:55.809496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809417    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-podinfo\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.809496 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809463    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-data\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.809713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809520    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-kube-api-access-vpgw9\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-kube-api-access-vpgw9\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.809713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809552    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-token\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-token\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.809713 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809672    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06d4305-2da3-41b4-9f80-62b8934617b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06d4305-2da3-41b4-9f80-62b8934617b3-istiod-ca-cert\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.809951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809767    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-envoy\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.809951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809781    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-data" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:55.809951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809836    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-credential-socket\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.809951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809897    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-certs\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.810190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.809995    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-socket\") pod \"e06d4305-2da3-41b4-9f80-62b8934617b3\" (UID: \"e06d4305-2da3-41b4-9f80-62b8934617b3\") "
Nov 03 11:32:55.810190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810147    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-credential-socket" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:55.810190 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810170    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06d4305-2da3-41b4-9f80-62b8934617b3-istiod-ca-cert" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:32:55.810534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810349    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-socket" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:55.810534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810364    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-certs" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:55.810651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810597    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06d4305-2da3-41b4-9f80-62b8934617b3-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:55.810651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810618    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:55.810651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810630    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:55.810651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810639    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:55.810651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.810648    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:55.811905 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.811863    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-podinfo" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:32:55.812101 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.812046    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-kube-api-access-vpgw9" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "kube-api-access-vpgw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:55.812390 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.812368    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-envoy" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:32:55.812489 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.812397    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-token" pod "e06d4305-2da3-41b4-9f80-62b8934617b3" (UID: "e06d4305-2da3-41b4-9f80-62b8934617b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:32:55.911981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.911912    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:55.911981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.911981    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpgw9\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-kube-api-access-vpgw9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:55.912197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.911994    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:55.912197 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:55.912008    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06d4305-2da3-41b4-9f80-62b8934617b3-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:32:56.230869 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.230834    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" path="/var/lib/kubelet/pods/4c7724d8-aa88-49c8-9a08-844b43f5a00d/volumes"
Nov 03 11:32:56.231251 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.231237    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941c0d96-a831-41d5-9392-7802117b42ad" path="/var/lib/kubelet/pods/941c0d96-a831-41d5-9392-7802117b42ad/volumes"
Nov 03 11:32:56.257876 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.257839    5393 generic.go:356] "Generic (PLEG): container finished" podID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerID="96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c" exitCode=0
Nov 03 11:32:56.258376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.257891    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"
Nov 03 11:32:56.258376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.257932    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" event={"ID":"e06d4305-2da3-41b4-9f80-62b8934617b3","Type":"ContainerDied","Data":"96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c"}
Nov 03 11:32:56.258376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.258005    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx" event={"ID":"e06d4305-2da3-41b4-9f80-62b8934617b3","Type":"ContainerDied","Data":"e2576e44969774ef01dd138650af73eed9a33e18405b119337c22ffa800e20e5"}
Nov 03 11:32:56.258376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.258040    5393 scope.go:122] "RemoveContainer" containerID="96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c"
Nov 03 11:32:56.268578 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.268550    5393 scope.go:122] "RemoveContainer" containerID="a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656"
Nov 03 11:32:56.277531 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.277493    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"]
Nov 03 11:32:56.279954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.279928    5393 scope.go:122] "RemoveContainer" containerID="96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c"
Nov 03 11:32:56.280296 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:56.280275    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c\": container with ID starting with 96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c not found: ID does not exist" containerID="96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c"
Nov 03 11:32:56.280384 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.280307    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c"} err="failed to get container status \"96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c\": rpc error: code = NotFound desc = could not find container \"96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c\": container with ID starting with 96a4250c470ceaaccb6828877601b9df601113118b944da729b6b6de0d7cd68c not found: ID does not exist"
Nov 03 11:32:56.280384 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.280326    5393 scope.go:122] "RemoveContainer" containerID="a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656"
Nov 03 11:32:56.280581 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:32:56.280565    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656\": container with ID starting with a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656 not found: ID does not exist" containerID="a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656"
Nov 03 11:32:56.280631 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.280586    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656"} err="failed to get container status \"a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656\": rpc error: code = NotFound desc = could not find container \"a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656\": container with ID starting with a05554be8bc4eb17117469fde46c63b677cb330b675950d0036996f5799d9656 not found: ID does not exist"
Nov 03 11:32:56.282362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:56.282341    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5711cf-openshift-default-7f5d5f45-vfqjx"]
Nov 03 11:32:57.769128 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:57.769094    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh" podUID="9d5239bc-ba6c-4d26-b548-232d9252f273" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:57.769516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:57.769107    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn" podUID="25c86b89-8fd8-499a-a6a5-6846dd17ac26" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 11:32:58.230046 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:32:58.230010    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" path="/var/lib/kubelet/pods/e06d4305-2da3-41b4-9f80-62b8934617b3/volumes"
Nov 03 11:33:03.072132 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072094    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"]
Nov 03 11:33:03.072626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072584    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy"
Nov 03 11:33:03.072626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072601    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy"
Nov 03 11:33:03.072626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072617    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy"
Nov 03 11:33:03.072626 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072623    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072653    5393 cpu_manager.go:414] "containerMap: removing container" podUID="941c0d96-a831-41d5-9392-7802117b42ad" containerName="payload-pre-processing"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072659    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941c0d96-a831-41d5-9392-7802117b42ad" containerName="payload-pre-processing"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072668    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2812357e-7c18-4d0f-afd6-b419cdefd21f" containerName="revoke-keys"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072674    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2812357e-7c18-4d0f-afd6-b419cdefd21f" containerName="revoke-keys"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072753    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6469e418-320e-4d01-86a4-99b169b7fc34" containerName="istio-proxy"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072765    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072774    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072785    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072792    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="941c0d96-a831-41d5-9392-7802117b42ad" containerName="payload-pre-processing"
Nov 03 11:33:03.072797 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.072797    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2812357e-7c18-4d0f-afd6-b419cdefd21f" containerName="revoke-keys"
Nov 03 11:33:03.094500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.094462    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"]
Nov 03 11:33:03.094500 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.094496    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.175172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.175132    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 11:33:03.175365 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.175191    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbt49\""
Nov 03 11:33:03.196954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.196876    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lfl\" (UniqueName: \"kubernetes.io/projected/4f4cacd8-86d2-477a-9ae9-679e05e0182d-kube-api-access-75lfl\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.196954 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.196946    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.197172 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.197110    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.299249 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.299203    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lfl\" (UniqueName: \"kubernetes.io/projected/4f4cacd8-86d2-477a-9ae9-679e05e0182d-kube-api-access-75lfl\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.299830 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.299801    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.300135 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.300116    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.303901 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.303875    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.320773 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.320737    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.324840 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.324730    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lfl\" (UniqueName: \"kubernetes.io/projected/4f4cacd8-86d2-477a-9ae9-679e05e0182d-kube-api-access-75lfl\") pod \"maas-api-revoke-sub-da340de4-2t59c\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.420195 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.420162    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:03.438239 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.438211    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"]
Nov 03 11:33:03.441711 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.441683    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce5555-cmdll"]
Nov 03 11:33:03.698761 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:03.698730    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"]
Nov 03 11:33:03.701319 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:33:03.701287    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4cacd8_86d2_477a_9ae9_679e05e0182d.slice/crio-5a02b857510868453a0257ffd42f7dfca1adbb5a6c170525d737075f5288f354 WatchSource:0}: Error finding container 5a02b857510868453a0257ffd42f7dfca1adbb5a6c170525d737075f5288f354: Status 404 returned error can't find the container with id 5a02b857510868453a0257ffd42f7dfca1adbb5a6c170525d737075f5288f354
Nov 03 11:33:04.230969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.230911    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2812357e-7c18-4d0f-afd6-b419cdefd21f" path="/var/lib/kubelet/pods/2812357e-7c18-4d0f-afd6-b419cdefd21f/volumes"
Nov 03 11:33:04.341579 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.341544    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c" event={"ID":"4f4cacd8-86d2-477a-9ae9-679e05e0182d","Type":"ContainerStarted","Data":"d44c204ccf348aaa4c5eaccf2cefa8c8dbaee424e0294a82aaabe954ce14ee40"}
Nov 03 11:33:04.341698 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.341606    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c" event={"ID":"4f4cacd8-86d2-477a-9ae9-679e05e0182d","Type":"ContainerStarted","Data":"5a02b857510868453a0257ffd42f7dfca1adbb5a6c170525d737075f5288f354"}
Nov 03 11:33:04.364141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.364091    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c" podStartSLOduration=1.36407787 podStartE2EDuration="1.36407787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:33:04.361582694 +0000 UTC m=+3828.266437137" watchObservedRunningTime="2026-11-03 11:33:04.36407787 +0000 UTC m=+3828.268932310"
Nov 03 11:33:04.688456 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.688428    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:33:04.691351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.691327    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:33:04.814117 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.814027    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c86b89-8fd8-499a-a6a5-6846dd17ac26-kube-api-access-v8scf\" (UniqueName: \"kubernetes.io/projected/25c86b89-8fd8-499a-a6a5-6846dd17ac26-kube-api-access-v8scf\") pod \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\" (UID: \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\") "
Nov 03 11:33:04.814117 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.814072    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5239bc-ba6c-4d26-b548-232d9252f273-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5239bc-ba6c-4d26-b548-232d9252f273-plugins-config-volume\") pod \"9d5239bc-ba6c-4d26-b548-232d9252f273\" (UID: \"9d5239bc-ba6c-4d26-b548-232d9252f273\") "
Nov 03 11:33:04.814117 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.814112    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c86b89-8fd8-499a-a6a5-6846dd17ac26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c86b89-8fd8-499a-a6a5-6846dd17ac26-plugins-config-volume\") pod \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\" (UID: \"25c86b89-8fd8-499a-a6a5-6846dd17ac26\") "
Nov 03 11:33:04.814407 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.814156    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5239bc-ba6c-4d26-b548-232d9252f273-kube-api-access-w5cdx\" (UniqueName: \"kubernetes.io/projected/9d5239bc-ba6c-4d26-b548-232d9252f273-kube-api-access-w5cdx\") pod \"9d5239bc-ba6c-4d26-b548-232d9252f273\" (UID: \"9d5239bc-ba6c-4d26-b548-232d9252f273\") "
Nov 03 11:33:04.814538 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.814514    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5239bc-ba6c-4d26-b548-232d9252f273-plugins-config-volume" pod "9d5239bc-ba6c-4d26-b548-232d9252f273" (UID: "9d5239bc-ba6c-4d26-b548-232d9252f273"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:04.814584 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.814514    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c86b89-8fd8-499a-a6a5-6846dd17ac26-plugins-config-volume" pod "25c86b89-8fd8-499a-a6a5-6846dd17ac26" (UID: "25c86b89-8fd8-499a-a6a5-6846dd17ac26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:04.814754 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.814734    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5239bc-ba6c-4d26-b548-232d9252f273-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:04.814874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.814754    5393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c86b89-8fd8-499a-a6a5-6846dd17ac26-plugins-config-volume\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:04.816534 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.816511    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c86b89-8fd8-499a-a6a5-6846dd17ac26-kube-api-access-v8scf" pod "25c86b89-8fd8-499a-a6a5-6846dd17ac26" (UID: "25c86b89-8fd8-499a-a6a5-6846dd17ac26"). InnerVolumeSpecName "kube-api-access-v8scf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:04.816640 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.816538    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5239bc-ba6c-4d26-b548-232d9252f273-kube-api-access-w5cdx" pod "9d5239bc-ba6c-4d26-b548-232d9252f273" (UID: "9d5239bc-ba6c-4d26-b548-232d9252f273"). InnerVolumeSpecName "kube-api-access-w5cdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:04.915947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.915883    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8scf\" (UniqueName: \"kubernetes.io/projected/25c86b89-8fd8-499a-a6a5-6846dd17ac26-kube-api-access-v8scf\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:04.915947 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:04.915943    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5cdx\" (UniqueName: \"kubernetes.io/projected/9d5239bc-ba6c-4d26-b548-232d9252f273-kube-api-access-w5cdx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:05.357359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.357316    5393 generic.go:356] "Generic (PLEG): container finished" podID="25c86b89-8fd8-499a-a6a5-6846dd17ac26" containerID="7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145" exitCode=0
Nov 03 11:33:05.357854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.357384    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"
Nov 03 11:33:05.357854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.357417    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn" event={"ID":"25c86b89-8fd8-499a-a6a5-6846dd17ac26","Type":"ContainerDied","Data":"7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145"}
Nov 03 11:33:05.357854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.357509    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn" event={"ID":"25c86b89-8fd8-499a-a6a5-6846dd17ac26","Type":"ContainerDied","Data":"548b815a72d6a24e6cdaf7b3cfb67387e52513b6c4f5cb126e96c907c08c229e"}
Nov 03 11:33:05.357854 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.357533    5393 scope.go:122] "RemoveContainer" containerID="7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145"
Nov 03 11:33:05.360117 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.360085    5393 generic.go:356] "Generic (PLEG): container finished" podID="4f4cacd8-86d2-477a-9ae9-679e05e0182d" containerID="d44c204ccf348aaa4c5eaccf2cefa8c8dbaee424e0294a82aaabe954ce14ee40" exitCode=0
Nov 03 11:33:05.360244 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.360165    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c" event={"ID":"4f4cacd8-86d2-477a-9ae9-679e05e0182d","Type":"ContainerDied","Data":"d44c204ccf348aaa4c5eaccf2cefa8c8dbaee424e0294a82aaabe954ce14ee40"}
Nov 03 11:33:05.361992 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.361969    5393 generic.go:356] "Generic (PLEG): container finished" podID="9d5239bc-ba6c-4d26-b548-232d9252f273" containerID="69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e" exitCode=0
Nov 03 11:33:05.362096 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.362030    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh" event={"ID":"9d5239bc-ba6c-4d26-b548-232d9252f273","Type":"ContainerDied","Data":"69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e"}
Nov 03 11:33:05.362096 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.362075    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh" event={"ID":"9d5239bc-ba6c-4d26-b548-232d9252f273","Type":"ContainerDied","Data":"141f843a6a1bfcaf746ea995eca0baece9177addcbe1973f39a0467a69a564b3"}
Nov 03 11:33:05.362226 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.362155    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"
Nov 03 11:33:05.369039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.368973    5393 scope.go:122] "RemoveContainer" containerID="7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145"
Nov 03 11:33:05.369262 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:05.369243    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145\": container with ID starting with 7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145 not found: ID does not exist" containerID="7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145"
Nov 03 11:33:05.369312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.369272    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145"} err="failed to get container status \"7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145\": rpc error: code = NotFound desc = could not find container \"7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145\": container with ID starting with 7974edb174611081f34329676f807ebc915b73e7b2547e0616d04262d43f7145 not found: ID does not exist"
Nov 03 11:33:05.369312 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.369291    5393 scope.go:122] "RemoveContainer" containerID="69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e"
Nov 03 11:33:05.379669 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.379650    5393 scope.go:122] "RemoveContainer" containerID="69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e"
Nov 03 11:33:05.380043 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:05.380015    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e\": container with ID starting with 69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e not found: ID does not exist" containerID="69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e"
Nov 03 11:33:05.380157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.380051    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e"} err="failed to get container status \"69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e\": rpc error: code = NotFound desc = could not find container \"69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e\": container with ID starting with 69b784799747e441c6dcb08abc8c2461e2d735137a39786ebc26701ce60f6f6e not found: ID does not exist"
Nov 03 11:33:05.392424 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.392402    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"]
Nov 03 11:33:05.396473 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.396450    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5711cf-794f89bc75-gndnn"]
Nov 03 11:33:05.408825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.408806    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"]
Nov 03 11:33:05.410741 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:05.410721    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5711cf-565c95b4c5-57tnh"]
Nov 03 11:33:06.229795 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.229764    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c86b89-8fd8-499a-a6a5-6846dd17ac26" path="/var/lib/kubelet/pods/25c86b89-8fd8-499a-a6a5-6846dd17ac26/volumes"
Nov 03 11:33:06.230192 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.230177    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5239bc-ba6c-4d26-b548-232d9252f273" path="/var/lib/kubelet/pods/9d5239bc-ba6c-4d26-b548-232d9252f273/volumes"
Nov 03 11:33:06.516681 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.516652    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:06.635379 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.635344    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4cacd8-86d2-477a-9ae9-679e05e0182d-kube-api-access-75lfl\" (UniqueName: \"kubernetes.io/projected/4f4cacd8-86d2-477a-9ae9-679e05e0182d-kube-api-access-75lfl\") pod \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") "
Nov 03 11:33:06.635559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.635420    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-openshift-ca\") pod \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") "
Nov 03 11:33:06.635559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.635456    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-xks-ca\") pod \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\" (UID: \"4f4cacd8-86d2-477a-9ae9-679e05e0182d\") "
Nov 03 11:33:06.640861 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.636689    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-openshift-ca" pod "4f4cacd8-86d2-477a-9ae9-679e05e0182d" (UID: "4f4cacd8-86d2-477a-9ae9-679e05e0182d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:06.660435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.660388    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-xks-ca" pod "4f4cacd8-86d2-477a-9ae9-679e05e0182d" (UID: "4f4cacd8-86d2-477a-9ae9-679e05e0182d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:33:06.660435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.660415    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4cacd8-86d2-477a-9ae9-679e05e0182d-kube-api-access-75lfl" pod "4f4cacd8-86d2-477a-9ae9-679e05e0182d" (UID: "4f4cacd8-86d2-477a-9ae9-679e05e0182d"). InnerVolumeSpecName "kube-api-access-75lfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:06.736512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.736466    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75lfl\" (UniqueName: \"kubernetes.io/projected/4f4cacd8-86d2-477a-9ae9-679e05e0182d-kube-api-access-75lfl\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:06.736512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.736504    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-openshift-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:06.736512 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:06.736517    5393 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4cacd8-86d2-477a-9ae9-679e05e0182d-maas-api-xks-ca\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:07.376819 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:07.376781    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c" event={"ID":"4f4cacd8-86d2-477a-9ae9-679e05e0182d","Type":"ContainerDied","Data":"5a02b857510868453a0257ffd42f7dfca1adbb5a6c170525d737075f5288f354"}
Nov 03 11:33:07.377030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:07.376831    5393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a02b857510868453a0257ffd42f7dfca1adbb5a6c170525d737075f5288f354"
Nov 03 11:33:07.377030 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:07.376797    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da340de4-2t59c"
Nov 03 11:33:09.938893 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:09.938852    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" containerID="cri-o://60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4" gracePeriod=45
Nov 03 11:33:09.946662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:09.946622    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main" containerID="cri-o://a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5" gracePeriod=45
Nov 03 11:33:09.966414 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:09.966374    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main" containerID="cri-o://cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6" gracePeriod=45
Nov 03 11:33:10.004680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.004647    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused"
Nov 03 11:33:10.268607 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.268573    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 11:33:10.272618 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.272589    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 11:33:10.312228 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.312199    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 11:33:10.380740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380508    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-home\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-home\") pod \"d21f137d-3ce4-49d8-9175-63b51e04f80c\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") "
Nov 03 11:33:10.380740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380568    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21f137d-3ce4-49d8-9175-63b51e04f80c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d21f137d-3ce4-49d8-9175-63b51e04f80c-tls-certs\") pod \"d21f137d-3ce4-49d8-9175-63b51e04f80c\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") "
Nov 03 11:33:10.380740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380601    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-dshm\") pod \"0fe8b503-1baf-4082-b14c-6ee94460567e\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") "
Nov 03 11:33:10.380740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380660    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe8b503-1baf-4082-b14c-6ee94460567e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe8b503-1baf-4082-b14c-6ee94460567e-tls-certs\") pod \"0fe8b503-1baf-4082-b14c-6ee94460567e\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") "
Nov 03 11:33:10.380740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380700    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21f137d-3ce4-49d8-9175-63b51e04f80c-kube-api-access-z9fzw\" (UniqueName: \"kubernetes.io/projected/d21f137d-3ce4-49d8-9175-63b51e04f80c-kube-api-access-z9fzw\") pod \"d21f137d-3ce4-49d8-9175-63b51e04f80c\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") "
Nov 03 11:33:10.380740 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380733    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-model-cache\") pod \"0fe8b503-1baf-4082-b14c-6ee94460567e\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") "
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380780    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-tmp-dir\") pod \"0fe8b503-1baf-4082-b14c-6ee94460567e\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") "
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380808    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-dshm\") pod \"d21f137d-3ce4-49d8-9175-63b51e04f80c\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") "
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380864    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-home\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-home\") pod \"0fe8b503-1baf-4082-b14c-6ee94460567e\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") "
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380938    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe8b503-1baf-4082-b14c-6ee94460567e-kube-api-access-qtrrj\" (UniqueName: \"kubernetes.io/projected/0fe8b503-1baf-4082-b14c-6ee94460567e-kube-api-access-qtrrj\") pod \"0fe8b503-1baf-4082-b14c-6ee94460567e\" (UID: \"0fe8b503-1baf-4082-b14c-6ee94460567e\") "
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.380986    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-tmp-dir\") pod \"d21f137d-3ce4-49d8-9175-63b51e04f80c\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") "
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.381056    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-model-cache\") pod \"d21f137d-3ce4-49d8-9175-63b51e04f80c\" (UID: \"d21f137d-3ce4-49d8-9175-63b51e04f80c\") "
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.381135    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-home" pod "d21f137d-3ce4-49d8-9175-63b51e04f80c" (UID: "d21f137d-3ce4-49d8-9175-63b51e04f80c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.381338    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-model-cache" pod "0fe8b503-1baf-4082-b14c-6ee94460567e" (UID: "0fe8b503-1baf-4082-b14c-6ee94460567e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.381572    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-model-cache" pod "d21f137d-3ce4-49d8-9175-63b51e04f80c" (UID: "d21f137d-3ce4-49d8-9175-63b51e04f80c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.381794    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-tmp-dir" pod "d21f137d-3ce4-49d8-9175-63b51e04f80c" (UID: "d21f137d-3ce4-49d8-9175-63b51e04f80c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.383371    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.383394    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.383409    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.381260 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.383425    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.384905 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.383619    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-home" pod "0fe8b503-1baf-4082-b14c-6ee94460567e" (UID: "0fe8b503-1baf-4082-b14c-6ee94460567e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.385245 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.385216    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-tmp-dir" pod "0fe8b503-1baf-4082-b14c-6ee94460567e" (UID: "0fe8b503-1baf-4082-b14c-6ee94460567e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.396868 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.396725    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-dshm" pod "0fe8b503-1baf-4082-b14c-6ee94460567e" (UID: "0fe8b503-1baf-4082-b14c-6ee94460567e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.397468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.397262    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21f137d-3ce4-49d8-9175-63b51e04f80c-tls-certs" pod "d21f137d-3ce4-49d8-9175-63b51e04f80c" (UID: "d21f137d-3ce4-49d8-9175-63b51e04f80c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:33:10.398430 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.398281    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe8b503-1baf-4082-b14c-6ee94460567e-tls-certs" pod "0fe8b503-1baf-4082-b14c-6ee94460567e" (UID: "0fe8b503-1baf-4082-b14c-6ee94460567e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:33:10.402103 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.402064    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-dshm" pod "d21f137d-3ce4-49d8-9175-63b51e04f80c" (UID: "d21f137d-3ce4-49d8-9175-63b51e04f80c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.402268 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.402127    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe8b503-1baf-4082-b14c-6ee94460567e-kube-api-access-qtrrj" pod "0fe8b503-1baf-4082-b14c-6ee94460567e" (UID: "0fe8b503-1baf-4082-b14c-6ee94460567e"). InnerVolumeSpecName "kube-api-access-qtrrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:10.406941 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.406114    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21f137d-3ce4-49d8-9175-63b51e04f80c-kube-api-access-z9fzw" pod "d21f137d-3ce4-49d8-9175-63b51e04f80c" (UID: "d21f137d-3ce4-49d8-9175-63b51e04f80c"). InnerVolumeSpecName "kube-api-access-z9fzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:10.416802 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.416767    5393 generic.go:356] "Generic (PLEG): container finished" podID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerID="60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4" exitCode=0
Nov 03 11:33:10.417232 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.417211    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"
Nov 03 11:33:10.417779 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.417308    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" event={"ID":"d21f137d-3ce4-49d8-9175-63b51e04f80c","Type":"ContainerDied","Data":"60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4"}
Nov 03 11:33:10.417885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.417806    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5" event={"ID":"d21f137d-3ce4-49d8-9175-63b51e04f80c","Type":"ContainerDied","Data":"8bd78084d21388e5bb1728687ea46618e8ccfd5368cccc6006a92785498424ec"}
Nov 03 11:33:10.417885 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.417836    5393 scope.go:122] "RemoveContainer" containerID="60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4"
Nov 03 11:33:10.422219 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.422184    5393 generic.go:356] "Generic (PLEG): container finished" podID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerID="a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5" exitCode=0
Nov 03 11:33:10.422356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.422336    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" event={"ID":"0fe8b503-1baf-4082-b14c-6ee94460567e","Type":"ContainerDied","Data":"a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5"}
Nov 03 11:33:10.422444 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.422382    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8" event={"ID":"0fe8b503-1baf-4082-b14c-6ee94460567e","Type":"ContainerDied","Data":"1ae9f9c7d270ad48048b8848a7e2a13ed1f699d69fc0f9d56bf0806eaeba824e"}
Nov 03 11:33:10.422507 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.422464    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"
Nov 03 11:33:10.431446 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.431413    5393 generic.go:356] "Generic (PLEG): container finished" podID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerID="cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6" exitCode=0
Nov 03 11:33:10.431595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.431488    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" event={"ID":"271be97a-c7bd-48fc-93c4-99e7a1de452a","Type":"ContainerDied","Data":"cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6"}
Nov 03 11:33:10.431595 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.431519    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk" event={"ID":"271be97a-c7bd-48fc-93c4-99e7a1de452a","Type":"ContainerDied","Data":"66a4ff43495f55a4e1f3b3f533e9673c616eeb4ce8263ecef34f68d223903639"}
Nov 03 11:33:10.431722 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.431598    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"
Nov 03 11:33:10.441499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.441468    5393 scope.go:122] "RemoveContainer" containerID="60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4"
Nov 03 11:33:10.441810 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:10.441791    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4\": container with ID starting with 60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4 not found: ID does not exist" containerID="60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4"
Nov 03 11:33:10.441863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.441819    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4"} err="failed to get container status \"60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4\": rpc error: code = NotFound desc = could not find container \"60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4\": container with ID starting with 60002e60e3fe164420fe6b20c2888b5443f7af6d4328c16a17f42c34a69cd3e4 not found: ID does not exist"
Nov 03 11:33:10.441863 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.441836    5393 scope.go:122] "RemoveContainer" containerID="a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5"
Nov 03 11:33:10.460806 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.460778    5393 scope.go:122] "RemoveContainer" containerID="a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5"
Nov 03 11:33:10.460909 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.460898    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"]
Nov 03 11:33:10.461267 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:10.461229    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5\": container with ID starting with a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5 not found: ID does not exist" containerID="a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5"
Nov 03 11:33:10.461458 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.461262    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5"} err="failed to get container status \"a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5\": rpc error: code = NotFound desc = could not find container \"a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5\": container with ID starting with a43c8493df03bcaf3ce80707dc74dec7713c698632b5e414b934c2123ca719e5 not found: ID does not exist"
Nov 03 11:33:10.461458 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.461286    5393 scope.go:122] "RemoveContainer" containerID="cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6"
Nov 03 11:33:10.464109 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.464081    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-2-w1-010232-kserve-8668d84b59-r62q5"]
Nov 03 11:33:10.470754 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.470726    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"]
Nov 03 11:33:10.472315 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.472294    5393 scope.go:122] "RemoveContainer" containerID="cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6"
Nov 03 11:33:10.473003 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:10.472976    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6\": container with ID starting with cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6 not found: ID does not exist" containerID="cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6"
Nov 03 11:33:10.473122 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.473010    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6"} err="failed to get container status \"cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6\": rpc error: code = NotFound desc = could not find container \"cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6\": container with ID starting with cf509ab1c67ca44dccbe104281d0c300dd18034118f8f5ddc0f8fc4ffebec8f6 not found: ID does not exist"
Nov 03 11:33:10.474660 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.474633    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-010232/distinct-w1-010232-kserve-6f46b7b64-hg7l8"]
Nov 03 11:33:10.484432 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.484352    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-home\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-home\") pod \"271be97a-c7bd-48fc-93c4-99e7a1de452a\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") "
Nov 03 11:33:10.484570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.484450    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-tmp-dir\") pod \"271be97a-c7bd-48fc-93c4-99e7a1de452a\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") "
Nov 03 11:33:10.484570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.484515    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-model-cache\") pod \"271be97a-c7bd-48fc-93c4-99e7a1de452a\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") "
Nov 03 11:33:10.484570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.484563    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271be97a-c7bd-48fc-93c4-99e7a1de452a-kube-api-access-c5lvr\" (UniqueName: \"kubernetes.io/projected/271be97a-c7bd-48fc-93c4-99e7a1de452a-kube-api-access-c5lvr\") pod \"271be97a-c7bd-48fc-93c4-99e7a1de452a\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") "
Nov 03 11:33:10.484766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.484598    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-home" pod "271be97a-c7bd-48fc-93c4-99e7a1de452a" (UID: "271be97a-c7bd-48fc-93c4-99e7a1de452a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.484766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.484649    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-dshm\") pod \"271be97a-c7bd-48fc-93c4-99e7a1de452a\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") "
Nov 03 11:33:10.484766 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.484690    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271be97a-c7bd-48fc-93c4-99e7a1de452a-tls-certs\" (UniqueName: \"kubernetes.io/secret/271be97a-c7bd-48fc-93c4-99e7a1de452a-tls-certs\") pod \"271be97a-c7bd-48fc-93c4-99e7a1de452a\" (UID: \"271be97a-c7bd-48fc-93c4-99e7a1de452a\") "
Nov 03 11:33:10.484957 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.484768    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-tmp-dir" pod "271be97a-c7bd-48fc-93c4-99e7a1de452a" (UID: "271be97a-c7bd-48fc-93c4-99e7a1de452a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.485091 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485069    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtrrj\" (UniqueName: \"kubernetes.io/projected/0fe8b503-1baf-4082-b14c-6ee94460567e-kube-api-access-qtrrj\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485094    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485109    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d21f137d-3ce4-49d8-9175-63b51e04f80c-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485122    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485135    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485151 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485149    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fe8b503-1baf-4082-b14c-6ee94460567e-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485162    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9fzw\" (UniqueName: \"kubernetes.io/projected/d21f137d-3ce4-49d8-9175-63b51e04f80c-kube-api-access-z9fzw\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485177    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485190    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d21f137d-3ce4-49d8-9175-63b51e04f80c-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485201    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fe8b503-1baf-4082-b14c-6ee94460567e-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.485588 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.485427    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-model-cache" pod "271be97a-c7bd-48fc-93c4-99e7a1de452a" (UID: "271be97a-c7bd-48fc-93c4-99e7a1de452a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.487083 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.487059    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271be97a-c7bd-48fc-93c4-99e7a1de452a-kube-api-access-c5lvr" pod "271be97a-c7bd-48fc-93c4-99e7a1de452a" (UID: "271be97a-c7bd-48fc-93c4-99e7a1de452a"). InnerVolumeSpecName "kube-api-access-c5lvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:10.487517 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.487494    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-dshm" pod "271be97a-c7bd-48fc-93c4-99e7a1de452a" (UID: "271be97a-c7bd-48fc-93c4-99e7a1de452a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:10.487570 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.487498    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271be97a-c7bd-48fc-93c4-99e7a1de452a-tls-certs" pod "271be97a-c7bd-48fc-93c4-99e7a1de452a" (UID: "271be97a-c7bd-48fc-93c4-99e7a1de452a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:33:10.586203 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.586165    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.586203 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.586205    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5lvr\" (UniqueName: \"kubernetes.io/projected/271be97a-c7bd-48fc-93c4-99e7a1de452a-kube-api-access-c5lvr\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.586427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.586219    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271be97a-c7bd-48fc-93c4-99e7a1de452a-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.586427 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.586233    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271be97a-c7bd-48fc-93c4-99e7a1de452a-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:10.621617 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.621575    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"]
Nov 03 11:33:10.621985 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.621953    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy" containerID="cri-o://988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af" gracePeriod=30
Nov 03 11:33:10.767622 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.767584    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"]
Nov 03 11:33:10.770838 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:10.770808    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-010232/simulator-w1-010232-kserve-88cb857c6-pqtlk"]
Nov 03 11:33:12.228551 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:12.228516    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" path="/var/lib/kubelet/pods/0fe8b503-1baf-4082-b14c-6ee94460567e/volumes"
Nov 03 11:33:12.228961 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:12.228946    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" path="/var/lib/kubelet/pods/271be97a-c7bd-48fc-93c4-99e7a1de452a/volumes"
Nov 03 11:33:12.229317 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:12.229304    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" path="/var/lib/kubelet/pods/d21f137d-3ce4-49d8-9175-63b51e04f80c/volumes"
Nov 03 11:33:13.188862 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:13.188822    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:33:15.877011 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:15.876985    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 11:33:16.051279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051187    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-envoy\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051284    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-podinfo\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051342    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-socket\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051382    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-kube-api-access-556h6\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-kube-api-access-556h6\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051416    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istiod-ca-cert\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051480 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051446    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-data\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051495    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-token\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-token\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051532    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-credential-socket\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051563    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-certs\") pod \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\" (UID: \"2e4bbcff-54aa-47e7-ae55-1e451c34a943\") "
Nov 03 11:33:16.051794 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051709    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-socket" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:16.052047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.051990    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-credential-socket" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:16.052276 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.052254    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.052392 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.052285    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.052392 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.052251    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-data" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:16.052515 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.052482    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-certs" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:16.052714 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.052651    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istiod-ca-cert" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:16.054250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.054092    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-podinfo" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:33:16.054369 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.054261    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-envoy" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:16.054596 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.054572    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-token" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:16.054717 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.054700    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-kube-api-access-556h6" pod "2e4bbcff-54aa-47e7-ae55-1e451c34a943" (UID: "2e4bbcff-54aa-47e7-ae55-1e451c34a943"). InnerVolumeSpecName "kube-api-access-556h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:16.152911 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.152878    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.152911 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.152908    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556h6\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-kube-api-access-556h6\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.153161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.152954    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.153161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.152966    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.153161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.152978    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.153161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.152991    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.153161 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.153003    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4bbcff-54aa-47e7-ae55-1e451c34a943-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:16.472021 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.471986    5393 generic.go:356] "Generic (PLEG): container finished" podID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerID="988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af" exitCode=0
Nov 03 11:33:16.472203 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.472055    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"
Nov 03 11:33:16.472203 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.472069    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" event={"ID":"2e4bbcff-54aa-47e7-ae55-1e451c34a943","Type":"ContainerDied","Data":"988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af"}
Nov 03 11:33:16.472203 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.472159    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk" event={"ID":"2e4bbcff-54aa-47e7-ae55-1e451c34a943","Type":"ContainerDied","Data":"4f37d6c023e2d02e9e30986b58ca7e7839c984041a61a20a66bcb345262de188"}
Nov 03 11:33:16.472203 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.472186    5393 scope.go:122] "RemoveContainer" containerID="988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af"
Nov 03 11:33:16.482051 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.482022    5393 scope.go:122] "RemoveContainer" containerID="781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da"
Nov 03 11:33:16.491668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.491643    5393 scope.go:122] "RemoveContainer" containerID="988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af"
Nov 03 11:33:16.492032 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:16.491998    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af\": container with ID starting with 988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af not found: ID does not exist" containerID="988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af"
Nov 03 11:33:16.492141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.492040    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af"} err="failed to get container status \"988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af\": rpc error: code = NotFound desc = could not find container \"988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af\": container with ID starting with 988d2d0c0634eb53a24e2c3fb40067dc2a9ad533cf606e968e493a9163cb70af not found: ID does not exist"
Nov 03 11:33:16.492141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.492058    5393 scope.go:122] "RemoveContainer" containerID="781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da"
Nov 03 11:33:16.492337 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:16.492316    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da\": container with ID starting with 781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da not found: ID does not exist" containerID="781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da"
Nov 03 11:33:16.492376 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.492344    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da"} err="failed to get container status \"781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da\": rpc error: code = NotFound desc = could not find container \"781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da\": container with ID starting with 781972cb18829072accafd415414a18c6badc48291fbce787cf6a019dc3335da not found: ID does not exist"
Nov 03 11:33:16.492677 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.492656    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"]
Nov 03 11:33:16.495131 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:16.495111    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ce5555-openshift-default-6fc8b94797-k7lfk"]
Nov 03 11:33:18.228981 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:18.228946    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" path="/var/lib/kubelet/pods/2e4bbcff-54aa-47e7-ae55-1e451c34a943/volumes"
Nov 03 11:33:41.351687 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:41.351638    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"]
Nov 03 11:33:41.352632 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:41.351978    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy" containerID="cri-o://32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0" gracePeriod=30
Nov 03 11:33:43.189046 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:43.189003    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:33:46.616289 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.616257    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 11:33:46.643820 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.643730    5393 generic.go:356] "Generic (PLEG): container finished" podID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerID="32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0" exitCode=0
Nov 03 11:33:46.644078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.643869    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" event={"ID":"dd4a2f72-5f48-432c-87c1-9a81cd568e12","Type":"ContainerDied","Data":"32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0"}
Nov 03 11:33:46.644078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.643958    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2" event={"ID":"dd4a2f72-5f48-432c-87c1-9a81cd568e12","Type":"ContainerDied","Data":"03a9d3e96b99399f90a7d01e55ce7dee943ac2f33fe23b858499d4475376fcdb"}
Nov 03 11:33:46.644078 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.643994    5393 scope.go:122] "RemoveContainer" containerID="32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0"
Nov 03 11:33:46.644303 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.643805    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"
Nov 03 11:33:46.655059 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.655040    5393 scope.go:122] "RemoveContainer" containerID="6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0"
Nov 03 11:33:46.664001 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.663981    5393 scope.go:122] "RemoveContainer" containerID="32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0"
Nov 03 11:33:46.664270 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:46.664253    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0\": container with ID starting with 32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0 not found: ID does not exist" containerID="32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0"
Nov 03 11:33:46.664331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.664283    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0"} err="failed to get container status \"32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0\": rpc error: code = NotFound desc = could not find container \"32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0\": container with ID starting with 32d4b65eb5124ca5f88bcbeedcda57249a6f411e2504223414fa2b87e9e6ddc0 not found: ID does not exist"
Nov 03 11:33:46.664331 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.664300    5393 scope.go:122] "RemoveContainer" containerID="6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0"
Nov 03 11:33:46.664577 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:33:46.664561    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0\": container with ID starting with 6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0 not found: ID does not exist" containerID="6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0"
Nov 03 11:33:46.664658 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.664580    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0"} err="failed to get container status \"6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0\": rpc error: code = NotFound desc = could not find container \"6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0\": container with ID starting with 6af7859542db7ba3baa82868a28f55638ed8528c19c76a25ce6e8ffaac8a73f0 not found: ID does not exist"
Nov 03 11:33:46.787691 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.787653    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-podinfo\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.787960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.787720    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-kube-api-access-rlr47\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-kube-api-access-rlr47\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.787960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.787763    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istiod-ca-cert\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.787960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.787784    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-envoy\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.787960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.787810    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-token\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-token\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.787960 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.787851    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-socket\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.788295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.788171    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-socket" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.788295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.788248    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istiod-ca-cert" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:33:46.788295 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.788284    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-certs\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.788448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.788328    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-data\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.788448 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.788410    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-credential-socket\") pod \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\" (UID: \"dd4a2f72-5f48-432c-87c1-9a81cd568e12\") "
Nov 03 11:33:46.788779 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.788752    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-certs" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.788983 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.788958    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-data" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.789039 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.789022    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.789079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.789047    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.789079 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.789062    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.789169 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.789149    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-credential-socket" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.790338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.790308    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-kube-api-access-rlr47" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "kube-api-access-rlr47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:46.790338 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.790310    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-podinfo" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:33:46.790723 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.790709    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-envoy" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:33:46.791100 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.791082    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-token" pod "dd4a2f72-5f48-432c-87c1-9a81cd568e12" (UID: "dd4a2f72-5f48-432c-87c1-9a81cd568e12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:33:46.850989 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.850951    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"]
Nov 03 11:33:46.867705 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.867674    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"]
Nov 03 11:33:46.889624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.889594    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.889624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.889623    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.889991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.889637    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlr47\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-kube-api-access-rlr47\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.889991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.889652    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.889991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.889666    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.889991 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.889678    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4a2f72-5f48-432c-87c1-9a81cd568e12-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:33:46.967666 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.967637    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"]
Nov 03 11:33:46.975417 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:46.975387    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-836589-openshift-default-54b68dc845-v44r2"]
Nov 03 11:33:48.229435 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:33:48.229400    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" path="/var/lib/kubelet/pods/dd4a2f72-5f48-432c-87c1-9a81cd568e12/volumes"
Nov 03 11:34:01.872014 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:01.871942    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main" containerID="cri-o://6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb" gracePeriod=45
Nov 03 11:34:01.888285 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:01.888231    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" containerID="cri-o://68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5" gracePeriod=45
Nov 03 11:34:02.027326 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.027251    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"]
Nov 03 11:34:02.035737 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.035702    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a11d4351-8nzvz"]
Nov 03 11:34:02.057472 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.057430    5393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": dial tcp 10.134.0.44:8000: connect: connection refused"
Nov 03 11:34:02.148345 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.148319    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 11:34:02.178811 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.178784    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 11:34:02.230554 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.230524    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3" path="/var/lib/kubelet/pods/bf942b2d-bf1e-4300-8fe7-ad6f602d7ce3/volumes"
Nov 03 11:34:02.235499 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235465    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea895738-0cc7-4285-997a-35097600664d-kube-api-access-xpln9\" (UniqueName: \"kubernetes.io/projected/ea895738-0cc7-4285-997a-35097600664d-kube-api-access-xpln9\") pod \"ea895738-0cc7-4285-997a-35097600664d\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") "
Nov 03 11:34:02.235648 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235507    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-dshm\") pod \"ea895738-0cc7-4285-997a-35097600664d\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") "
Nov 03 11:34:02.235703 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235668    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-home\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-home\") pod \"02beb4c4-a80f-493a-a421-ac714c3e256d\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") "
Nov 03 11:34:02.235768 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235727    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-tmp-dir\") pod \"02beb4c4-a80f-493a-a421-ac714c3e256d\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") "
Nov 03 11:34:02.235831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235772    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-model-cache\") pod \"ea895738-0cc7-4285-997a-35097600664d\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") "
Nov 03 11:34:02.235899 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235839    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-dshm\") pod \"02beb4c4-a80f-493a-a421-ac714c3e256d\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") "
Nov 03 11:34:02.235899 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235881    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-model-cache\") pod \"02beb4c4-a80f-493a-a421-ac714c3e256d\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") "
Nov 03 11:34:02.236041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235975    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea895738-0cc7-4285-997a-35097600664d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea895738-0cc7-4285-997a-35097600664d-tls-certs\") pod \"ea895738-0cc7-4285-997a-35097600664d\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") "
Nov 03 11:34:02.236041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.235983    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-home" pod "02beb4c4-a80f-493a-a421-ac714c3e256d" (UID: "02beb4c4-a80f-493a-a421-ac714c3e256d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.236041 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236016    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02beb4c4-a80f-493a-a421-ac714c3e256d-tls-certs\" (UniqueName: \"kubernetes.io/secret/02beb4c4-a80f-493a-a421-ac714c3e256d-tls-certs\") pod \"02beb4c4-a80f-493a-a421-ac714c3e256d\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") "
Nov 03 11:34:02.236255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236069    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-tmp-dir" pod "02beb4c4-a80f-493a-a421-ac714c3e256d" (UID: "02beb4c4-a80f-493a-a421-ac714c3e256d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.236255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236085    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-home\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-home\") pod \"ea895738-0cc7-4285-997a-35097600664d\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") "
Nov 03 11:34:02.236255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236121    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02beb4c4-a80f-493a-a421-ac714c3e256d-kube-api-access-qd7gx\" (UniqueName: \"kubernetes.io/projected/02beb4c4-a80f-493a-a421-ac714c3e256d-kube-api-access-qd7gx\") pod \"02beb4c4-a80f-493a-a421-ac714c3e256d\" (UID: \"02beb4c4-a80f-493a-a421-ac714c3e256d\") "
Nov 03 11:34:02.236255 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236167    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-tmp-dir\") pod \"ea895738-0cc7-4285-997a-35097600664d\" (UID: \"ea895738-0cc7-4285-997a-35097600664d\") "
Nov 03 11:34:02.236516 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236499    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.236624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236521    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.236624 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236582    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-model-cache" pod "ea895738-0cc7-4285-997a-35097600664d" (UID: "ea895738-0cc7-4285-997a-35097600664d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.236834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236730    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-tmp-dir" pod "ea895738-0cc7-4285-997a-35097600664d" (UID: "ea895738-0cc7-4285-997a-35097600664d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.236834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.236792    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-home" pod "ea895738-0cc7-4285-997a-35097600664d" (UID: "ea895738-0cc7-4285-997a-35097600664d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.237147 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.237119    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-model-cache" pod "02beb4c4-a80f-493a-a421-ac714c3e256d" (UID: "02beb4c4-a80f-493a-a421-ac714c3e256d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.238441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.238408    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-dshm" pod "ea895738-0cc7-4285-997a-35097600664d" (UID: "ea895738-0cc7-4285-997a-35097600664d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.238559 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.238508    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea895738-0cc7-4285-997a-35097600664d-kube-api-access-xpln9" pod "ea895738-0cc7-4285-997a-35097600664d" (UID: "ea895738-0cc7-4285-997a-35097600664d"). InnerVolumeSpecName "kube-api-access-xpln9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:02.239296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.239269    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02beb4c4-a80f-493a-a421-ac714c3e256d-kube-api-access-qd7gx" pod "02beb4c4-a80f-493a-a421-ac714c3e256d" (UID: "02beb4c4-a80f-493a-a421-ac714c3e256d"). InnerVolumeSpecName "kube-api-access-qd7gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:02.239478 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.239454    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02beb4c4-a80f-493a-a421-ac714c3e256d-tls-certs" pod "02beb4c4-a80f-493a-a421-ac714c3e256d" (UID: "02beb4c4-a80f-493a-a421-ac714c3e256d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:34:02.239765 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.239744    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea895738-0cc7-4285-997a-35097600664d-tls-certs" pod "ea895738-0cc7-4285-997a-35097600664d" (UID: "ea895738-0cc7-4285-997a-35097600664d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:34:02.241181 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.241165    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-dshm" pod "02beb4c4-a80f-493a-a421-ac714c3e256d" (UID: "02beb4c4-a80f-493a-a421-ac714c3e256d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:02.337386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337347    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpln9\" (UniqueName: \"kubernetes.io/projected/ea895738-0cc7-4285-997a-35097600664d-kube-api-access-xpln9\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337378    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337388    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337386 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337397    5393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-dshm\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337405    5393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02beb4c4-a80f-493a-a421-ac714c3e256d-model-cache\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337413    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea895738-0cc7-4285-997a-35097600664d-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337423    5393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02beb4c4-a80f-493a-a421-ac714c3e256d-tls-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337431    5393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-home\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337438    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd7gx\" (UniqueName: \"kubernetes.io/projected/02beb4c4-a80f-493a-a421-ac714c3e256d-kube-api-access-qd7gx\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.337686 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.337446    5393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea895738-0cc7-4285-997a-35097600664d-tmp-dir\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:02.725884 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.725846    5393 generic.go:356] "Generic (PLEG): container finished" podID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerID="68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5" exitCode=0
Nov 03 11:34:02.726121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.725897    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"
Nov 03 11:34:02.726121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.725941    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" event={"ID":"02beb4c4-a80f-493a-a421-ac714c3e256d","Type":"ContainerDied","Data":"68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5"}
Nov 03 11:34:02.726121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.725984    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w" event={"ID":"02beb4c4-a80f-493a-a421-ac714c3e256d","Type":"ContainerDied","Data":"98eb41feb9d729cab1bf4b614407b9120ab95db07df8e74bf61bae82502cebed"}
Nov 03 11:34:02.726121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.726004    5393 scope.go:122] "RemoveContainer" containerID="68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5"
Nov 03 11:34:02.727263 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.727232    5393 generic.go:356] "Generic (PLEG): container finished" podID="ea895738-0cc7-4285-997a-35097600664d" containerID="6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb" exitCode=0
Nov 03 11:34:02.727356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.727264    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" event={"ID":"ea895738-0cc7-4285-997a-35097600664d","Type":"ContainerDied","Data":"6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb"}
Nov 03 11:34:02.727356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.727280    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"
Nov 03 11:34:02.727356 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.727295    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4" event={"ID":"ea895738-0cc7-4285-997a-35097600664d","Type":"ContainerDied","Data":"d215a36c9f102418ea01adf12d20e74c527f1754a14699732c32e51120720cdd"}
Nov 03 11:34:02.739453 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.739362    5393 scope.go:122] "RemoveContainer" containerID="68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5"
Nov 03 11:34:02.739761 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:34:02.739737    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5\": container with ID starting with 68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5 not found: ID does not exist" containerID="68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5"
Nov 03 11:34:02.739825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.739775    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5"} err="failed to get container status \"68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5\": rpc error: code = NotFound desc = could not find container \"68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5\": container with ID starting with 68be2d37a49621620f4b921ad77727fc658f499beaf85438aa33a61cea1006b5 not found: ID does not exist"
Nov 03 11:34:02.739825 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.739799    5393 scope.go:122] "RemoveContainer" containerID="6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb"
Nov 03 11:34:02.748350 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.748329    5393 scope.go:122] "RemoveContainer" containerID="6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb"
Nov 03 11:34:02.748619 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:34:02.748596    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb\": container with ID starting with 6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb not found: ID does not exist" containerID="6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb"
Nov 03 11:34:02.748684 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.748626    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb"} err="failed to get container status \"6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb\": rpc error: code = NotFound desc = could not find container \"6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb\": container with ID starting with 6a33a1d042f315313ae83a9dd763722044f2f20ce3076b67ff090a914fac97eb not found: ID does not exist"
Nov 03 11:34:02.752247 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.752226    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"]
Nov 03 11:34:02.756279 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.756260    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/premium-w3-b1c6f6-kserve-b8fbcf987-4z44w"]
Nov 03 11:34:02.767259 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.767237    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"]
Nov 03 11:34:02.768877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:02.768854    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b1c6f6/distinct-2-w3-b1c6f6-kserve-6fccb87c97-282d4"]
Nov 03 11:34:04.232141 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:04.232103    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" path="/var/lib/kubelet/pods/02beb4c4-a80f-493a-a421-ac714c3e256d/volumes"
Nov 03 11:34:04.232671 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:04.232605    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea895738-0cc7-4285-997a-35097600664d" path="/var/lib/kubelet/pods/ea895738-0cc7-4285-997a-35097600664d/volumes"
Nov 03 11:34:10.318084 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:10.318039    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"]
Nov 03 11:34:10.318641 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:10.318308    5393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy" containerID="cri-o://3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8" gracePeriod=30
Nov 03 11:34:13.188307 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:34:13.188265    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:34:14.042452 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:14.042426    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xmpx9_ff61cb9b-f039-48cf-8879-d73eb8ff0a4a/manager/0.log"
Nov 03 11:34:14.441397 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:14.441361    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/1.log"
Nov 03 11:34:14.930073 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:14.930037    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6fdb79d597-9n4gw_30b7fb6a-bf01-4e56-b60e-dd62207d919c/manager/0.log"
Nov 03 11:34:15.570651 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.570624    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 11:34:15.674441 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674403    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-token\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.674662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674451    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-podinfo\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.674662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674472    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-kube-api-access-ckcdq\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-kube-api-access-ckcdq\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.674662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674499    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-credential-socket\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.674662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674517    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istiod-ca-cert\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.674662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674579    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-certs\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.674662 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674616    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-socket\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.675058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674684    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-envoy\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.675058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674720    5393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-data\") pod \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\" (UID: \"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226\") "
Nov 03 11:34:15.675058 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.674902    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-certs" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.675238 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.675039    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istiod-ca-cert" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:34:15.675238 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.675136    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-socket" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.675238 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.675139    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-data" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.675238 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.675189    5393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-certs\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.675238 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.675227    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-credential-socket" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.677081 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.677058    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-kube-api-access-ckcdq" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "kube-api-access-ckcdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:15.677189 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.677175    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-envoy" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:34:15.677298 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.677283    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-podinfo" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 11:34:15.677393 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.677376    5393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-token" pod "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" (UID: "ed9a3ed4-56b5-48cc-91ad-3bdc09cff226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:34:15.776635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.776592    5393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-envoy\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.776635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.776624    5393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-data\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.776635 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.776638    5393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-token\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.776896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.776650    5393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istio-podinfo\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.776896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.776664    5393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckcdq\" (UniqueName: \"kubernetes.io/projected/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-kube-api-access-ckcdq\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.776896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.776677    5393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-credential-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.776896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.776689    5393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-istiod-ca-cert\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.776896 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.776701    5393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226-workload-socket\") on node \"ip-10-0-134-12.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 11:34:15.797316 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.797281    5393 generic.go:356] "Generic (PLEG): container finished" podID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerID="3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8" exitCode=0
Nov 03 11:34:15.797468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.797330    5393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"
Nov 03 11:34:15.797468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.797372    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" event={"ID":"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226","Type":"ContainerDied","Data":"3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8"}
Nov 03 11:34:15.797468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.797425    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg" event={"ID":"ed9a3ed4-56b5-48cc-91ad-3bdc09cff226","Type":"ContainerDied","Data":"152539bfbffab0ddd953a9023b51eea74c6304b9224bd326a82264cb3a44894e"}
Nov 03 11:34:15.797468 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.797450    5393 scope.go:122] "RemoveContainer" containerID="3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8"
Nov 03 11:34:15.807017 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.807001    5393 scope.go:122] "RemoveContainer" containerID="3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8"
Nov 03 11:34:15.807276 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:34:15.807260    5393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8\": container with ID starting with 3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8 not found: ID does not exist" containerID="3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8"
Nov 03 11:34:15.807322 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.807287    5393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8"} err="failed to get container status \"3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8\": rpc error: code = NotFound desc = could not find container \"3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8\": container with ID starting with 3e23444bfc20756ddf5783e619f463a77683defcffee66c0816471a50a1e39b8 not found: ID does not exist"
Nov 03 11:34:15.819708 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.819683    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"]
Nov 03 11:34:15.822951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.822902    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b1c6f6-openshift-default-5fcd7689dc-2rvkg"]
Nov 03 11:34:15.910874 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:15.910849    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/0.log"
Nov 03 11:34:16.236363 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:16.236328    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" path="/var/lib/kubelet/pods/ed9a3ed4-56b5-48cc-91ad-3bdc09cff226/volumes"
Nov 03 11:34:18.610163 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:18.610130    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:34:18.621665 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:18.621643    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6t57h_71f6bbd9-acd8-4534-8adb-ca7cd139d383/ovn-acl-logging/0.log"
Nov 03 11:34:26.018638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:26.018597    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"]
Nov 03 11:34:26.021121 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:26.021095    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-604aa2a3-x6n7p"]
Nov 03 11:34:26.228719 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:26.228682    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08db93e1-7c7f-4bf7-9042-4bccf397b914" path="/var/lib/kubelet/pods/08db93e1-7c7f-4bf7-9042-4bccf397b914/volumes"
Nov 03 11:34:43.187568 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:34:43.187525    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:34:47.891729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:47.891703    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xmpx9_ff61cb9b-f039-48cf-8879-d73eb8ff0a4a/manager/0.log"
Nov 03 11:34:48.309343 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:48.309257    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/1.log"
Nov 03 11:34:48.798898 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:48.798860    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6fdb79d597-9n4gw_30b7fb6a-bf01-4e56-b60e-dd62207d919c/manager/0.log"
Nov 03 11:34:49.825200 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:49.825166    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/0.log"
Nov 03 11:34:50.409775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:50.409743    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55d5b7b94d-v7gmn_7ff27b47-b606-427c-ad74-87c4d5d0bd43/maas-api/0.log"
Nov 03 11:34:51.526038 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:51.526010    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6869004a-cs4w7_b03af0f2-633e-48b4-ac32-f78749a60da3/revoke-keys/0.log"
Nov 03 11:34:52.315668 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:52.315638    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-da340de4-2t59c_4f4cacd8-86d2-477a-9ae9-679e05e0182d/revoke-keys/0.log"
Nov 03 11:34:52.506252 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:52.506211    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dc0fdf6e-bs96f_75b67e08-c92f-415c-a970-d9bb0a67eb79/revoke-keys/0.log"
Nov 03 11:34:53.238614 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:53.238580    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-dzp47_32224c3b-fe76-434f-8b26-a07d4419b2f8/postgres/0.log"
Nov 03 11:34:57.274032 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:57.273990    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-xm7qt_b5877c92-09b4-4dff-ad52-38660d7345fe/discovery/0.log"
Nov 03 11:34:57.646758 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:57.646730    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb777df86-kcgc4_e3236943-9a1b-4a62-8992-fd40416c20df/kube-auth-proxy/0.log"
Nov 03 11:34:58.035824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:58.035736    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-74hrv_55911ac4-1644-448a-bf1e-056207cdb3b9/istio-proxy/0.log"
Nov 03 11:34:58.434405 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:58.434376    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-fv94c_56fd579f-dff9-4b92-ac15-00bdf72f86a8/istio-proxy/0.log"
Nov 03 11:34:58.624593 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:34:58.624568    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hs9mc_7afc9a85-df4f-4401-8656-19750d21605a/istio-proxy/0.log"
Nov 03 11:35:01.020940 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:01.020890    5393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"]
Nov 03 11:35:01.025597 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:01.025568    5393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6869004a-cs4w7"]
Nov 03 11:35:02.228633 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:02.228597    5393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03af0f2-633e-48b4-ac32-f78749a60da3" path="/var/lib/kubelet/pods/b03af0f2-633e-48b4-ac32-f78749a60da3/volumes"
Nov 03 11:35:13.187833 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:35:13.187793    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:35:21.738371 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:21.738333    5393 scope.go:122] "RemoveContainer" containerID="71b583eb268847a66e0e1ef9e55f51072a7ac96cc188657c600dd2652dc5bfaa"
Nov 03 11:35:26.840829 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:26.840790    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w4txd_430e1cec-2732-4742-8eed-d2ba0fc4caf3/global-pull-secret-syncer/0.log"
Nov 03 11:35:27.151551 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:27.151512    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9d2nl_88501d15-b241-4af1-af7a-eabc4c417a08/konnectivity-agent/0.log"
Nov 03 11:35:27.216068 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:27.216006    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-12.us-east-2.compute.internal_3041328657ebd4fdb2a0f070aa3e29b6/haproxy/0.log"
Nov 03 11:35:37.522348 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.522321    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c37a9918-badd-4dc3-b824-58b547004e22/alertmanager/0.log"
Nov 03 11:35:37.548769 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.548742    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c37a9918-badd-4dc3-b824-58b547004e22/config-reloader/0.log"
Nov 03 11:35:37.569966 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.569912    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c37a9918-badd-4dc3-b824-58b547004e22/kube-rbac-proxy-web/0.log"
Nov 03 11:35:37.591359 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.591331    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c37a9918-badd-4dc3-b824-58b547004e22/kube-rbac-proxy/0.log"
Nov 03 11:35:37.613164 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.613136    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c37a9918-badd-4dc3-b824-58b547004e22/kube-rbac-proxy-metric/0.log"
Nov 03 11:35:37.634138 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.634111    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c37a9918-badd-4dc3-b824-58b547004e22/prom-label-proxy/0.log"
Nov 03 11:35:37.655969 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.655945    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c37a9918-badd-4dc3-b824-58b547004e22/init-config-reloader/0.log"
Nov 03 11:35:37.814300 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.814228    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-767d4fbf8c-86qwm_11b7246b-031f-46cd-ab96-814d2e5d4a72/metrics-server/0.log"
Nov 03 11:35:37.839834 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.839807    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-ttks6_f9634e11-0c6c-4832-95a9-6665d679e1fd/monitoring-plugin/0.log"
Nov 03 11:35:37.871325 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.871297    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78k5n_d16fcb39-bfc9-4d51-af89-14d83a3d07a1/node-exporter/0.log"
Nov 03 11:35:37.892850 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.892820    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78k5n_d16fcb39-bfc9-4d51-af89-14d83a3d07a1/kube-rbac-proxy/0.log"
Nov 03 11:35:37.914072 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:37.914051    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78k5n_d16fcb39-bfc9-4d51-af89-14d83a3d07a1/init-textfile/0.log"
Nov 03 11:35:38.082452 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082373    5393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"]
Nov 03 11:35:38.082781 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082767    5393 cpu_manager.go:414] "containerMap: removing container" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main"
Nov 03 11:35:38.082831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082783    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main"
Nov 03 11:35:38.082831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082796    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy"
Nov 03 11:35:38.082831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082801    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy"
Nov 03 11:35:38.082831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082809    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy"
Nov 03 11:35:38.082831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082814    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy"
Nov 03 11:35:38.082831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082824    5393 cpu_manager.go:414] "containerMap: removing container" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main"
Nov 03 11:35:38.082831 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082829    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082836    5393 cpu_manager.go:414] "containerMap: removing container" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082841    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06d4305-2da3-41b4-9f80-62b8934617b3" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082849    5393 cpu_manager.go:414] "containerMap: removing container" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082853    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082858    5393 cpu_manager.go:414] "containerMap: removing container" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082863    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082872    5393 cpu_manager.go:414] "containerMap: removing container" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082876    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082883    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082887    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082900    5393 cpu_manager.go:414] "containerMap: removing container" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082904    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082931    5393 cpu_manager.go:414] "containerMap: removing container" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082938    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082946    5393 cpu_manager.go:414] "containerMap: removing container" podUID="25c86b89-8fd8-499a-a6a5-6846dd17ac26" containerName="payload-pre-processing"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082953    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c86b89-8fd8-499a-a6a5-6846dd17ac26" containerName="payload-pre-processing"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082964    5393 cpu_manager.go:414] "containerMap: removing container" podUID="9d5239bc-ba6c-4d26-b548-232d9252f273" containerName="payload-processing"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082968    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5239bc-ba6c-4d26-b548-232d9252f273" containerName="payload-processing"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082975    5393 cpu_manager.go:414] "containerMap: removing container" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082980    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082988    5393 cpu_manager.go:414] "containerMap: removing container" podUID="4f4cacd8-86d2-477a-9ae9-679e05e0182d" containerName="revoke-keys"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.082992    5393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4cacd8-86d2-477a-9ae9-679e05e0182d" containerName="revoke-keys"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083054    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083062    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c86b89-8fd8-499a-a6a5-6846dd17ac26" containerName="payload-pre-processing"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083068    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5239bc-ba6c-4d26-b548-232d9252f273" containerName="payload-processing"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083074    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4bbcff-54aa-47e7-ae55-1e451c34a943" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083079    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4cacd8-86d2-477a-9ae9-679e05e0182d" containerName="revoke-keys"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083088    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="02beb4c4-a80f-493a-a421-ac714c3e256d" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083092    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9a3ed4-56b5-48cc-91ad-3bdc09cff226" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083097    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7724d8-aa88-49c8-9a08-844b43f5a00d" containerName="istio-proxy"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083103    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea895738-0cc7-4285-997a-35097600664d" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083108    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21f137d-3ce4-49d8-9175-63b51e04f80c" containerName="main"
Nov 03 11:35:38.083098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083113    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4a2f72-5f48-432c-87c1-9a81cd568e12" containerName="istio-proxy"
Nov 03 11:35:38.084387 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083118    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="271be97a-c7bd-48fc-93c4-99e7a1de452a" containerName="main"
Nov 03 11:35:38.084387 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.083125    5393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe8b503-1baf-4082-b14c-6ee94460567e" containerName="main"
Nov 03 11:35:38.086419 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.086400    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.095150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.094822    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v48x6\"/\"kube-root-ca.crt\""
Nov 03 11:35:38.095150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.094897    5393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v48x6\"/\"default-dockercfg-25c4q\""
Nov 03 11:35:38.095150 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.094996    5393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v48x6\"/\"openshift-service-ca.crt\""
Nov 03 11:35:38.096202 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.096184    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"]
Nov 03 11:35:38.155351 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.155315    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvc7\" (UniqueName: \"kubernetes.io/projected/a64036a6-2a43-4eae-952d-55895924e375-kube-api-access-qtvc7\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.155532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.155365    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-proc\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.155532 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.155477    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-podres\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.155609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.155532    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-lib-modules\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.155609 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.155586    5393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-sys\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.256671 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256635    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-lib-modules\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.256877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256686    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-sys\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.256877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256736    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvc7\" (UniqueName: \"kubernetes.io/projected/a64036a6-2a43-4eae-952d-55895924e375-kube-api-access-qtvc7\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.256877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256770    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-lib-modules\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.256877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256775    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-proc\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.256877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256824    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-proc\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.256877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256827    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-sys\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.256877 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256871    5393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-podres\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.257157 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.256997    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a64036a6-2a43-4eae-952d-55895924e375-podres\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.264657 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.264624    5393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvc7\" (UniqueName: \"kubernetes.io/projected/a64036a6-2a43-4eae-952d-55895924e375-kube-api-access-qtvc7\") pod \"perf-node-gather-daemonset-jvh5g\" (UID: \"a64036a6-2a43-4eae-952d-55895924e375\") " pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.399445 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.399407    5393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:38.490258 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.490224    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d7db6cd45-cj2lf_e88024db-34c5-4f34-b838-79feb2aa83fa/telemeter-client/0.log"
Nov 03 11:35:38.513362 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.513327    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d7db6cd45-cj2lf_e88024db-34c5-4f34-b838-79feb2aa83fa/reload/0.log"
Nov 03 11:35:38.542321 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.542297    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d7db6cd45-cj2lf_e88024db-34c5-4f34-b838-79feb2aa83fa/kube-rbac-proxy/0.log"
Nov 03 11:35:38.575535 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.575504    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b7876cf5-7mrjf_54b87353-bd2b-4a31-a5d3-9aa7582e0354/thanos-query/0.log"
Nov 03 11:35:38.601487 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.601464    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b7876cf5-7mrjf_54b87353-bd2b-4a31-a5d3-9aa7582e0354/kube-rbac-proxy-web/0.log"
Nov 03 11:35:38.625513 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.625484    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b7876cf5-7mrjf_54b87353-bd2b-4a31-a5d3-9aa7582e0354/kube-rbac-proxy/0.log"
Nov 03 11:35:38.651733 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.651660    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b7876cf5-7mrjf_54b87353-bd2b-4a31-a5d3-9aa7582e0354/prom-label-proxy/0.log"
Nov 03 11:35:38.671951 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.671912    5393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"]
Nov 03 11:35:38.674418 ip-10-0-134-12 kubenswrapper[5393]: W1103 11:35:38.674392    5393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda64036a6_2a43_4eae_952d_55895924e375.slice/crio-6644a45e10f1ba79226472a2725f51b8ae6b6fc8eaef392578a1736ef40caceb WatchSource:0}: Error finding container 6644a45e10f1ba79226472a2725f51b8ae6b6fc8eaef392578a1736ef40caceb: Status 404 returned error can't find the container with id 6644a45e10f1ba79226472a2725f51b8ae6b6fc8eaef392578a1736ef40caceb
Nov 03 11:35:38.680106 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.680082    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b7876cf5-7mrjf_54b87353-bd2b-4a31-a5d3-9aa7582e0354/kube-rbac-proxy-rules/0.log"
Nov 03 11:35:38.708725 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:38.708704    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76b7876cf5-7mrjf_54b87353-bd2b-4a31-a5d3-9aa7582e0354/kube-rbac-proxy-metrics/0.log"
Nov 03 11:35:39.197824 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:39.197787    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g" event={"ID":"a64036a6-2a43-4eae-952d-55895924e375","Type":"ContainerStarted","Data":"2a06baa59e544ff03c7ee746cd1bba420d3d6e75c9297bdd64b8b24497c2bcfd"}
Nov 03 11:35:39.198047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:39.197850    5393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g" event={"ID":"a64036a6-2a43-4eae-952d-55895924e375","Type":"ContainerStarted","Data":"6644a45e10f1ba79226472a2725f51b8ae6b6fc8eaef392578a1736ef40caceb"}
Nov 03 11:35:39.198047 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:39.197879    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:39.214970 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:39.214904    5393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g" podStartSLOduration=1.214890894 podStartE2EDuration="1.214890894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:35:39.212809974 +0000 UTC m=+3983.117664415" watchObservedRunningTime="2026-11-03 11:35:39.214890894 +0000 UTC m=+3983.119745329"
Nov 03 11:35:41.278296 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:41.278261    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7dd74748b9-c9n62_32e9ac83-284d-4d25-bb73-771438efc6c5/console/0.log"
Nov 03 11:35:41.652391 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:41.652360    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xmpx9_ff61cb9b-f039-48cf-8879-d73eb8ff0a4a/manager/0.log"
Nov 03 11:35:41.721346 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:41.721320    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/0.log"
Nov 03 11:35:42.506619 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:42.506574    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/1.log"
Nov 03 11:35:42.846778 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:42.846742    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6fdb79d597-9n4gw_30b7fb6a-bf01-4e56-b60e-dd62207d919c/manager/0.log"
Nov 03 11:35:43.187599 ip-10-0-134-12 kubenswrapper[5393]: E1103 11:35:43.187508    5393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-98krd" containerName="extractor"
Nov 03 11:35:43.533094 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:43.533009    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9zqgx_20cf9d6c-4dc2-49e0-93c9-d6115eeeb905/dns/0.log"
Nov 03 11:35:43.553526 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:43.553494    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9zqgx_20cf9d6c-4dc2-49e0-93c9-d6115eeeb905/kube-rbac-proxy/0.log"
Nov 03 11:35:43.735112 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:43.735082    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-crz98_51410712-fd08-4429-8d93-c0266660f335/dns-node-resolver/0.log"
Nov 03 11:35:44.604545 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:44.604516    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-67pqq_7532f5ee-e2ed-40a2-9d9e-277a392e1c9f/node-ca/0.log"
Nov 03 11:35:45.213680 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:45.213652    5393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v48x6/perf-node-gather-daemonset-jvh5g"
Nov 03 11:35:45.319431 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:45.319392    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xmpx9_ff61cb9b-f039-48cf-8879-d73eb8ff0a4a/manager/0.log"
Nov 03 11:35:45.391638 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:45.391610    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/0.log"
Nov 03 11:35:46.190729 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:46.190687    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/1.log"
Nov 03 11:35:46.579524 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:46.579437    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6fdb79d597-9n4gw_30b7fb6a-bf01-4e56-b60e-dd62207d919c/manager/0.log"
Nov 03 11:35:50.065055 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:50.065021    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xmpx9_ff61cb9b-f039-48cf-8879-d73eb8ff0a4a/manager/0.log"
Nov 03 11:35:50.143250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:50.143195    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/0.log"
Nov 03 11:35:50.788304 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:50.788258    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-xm7qt_b5877c92-09b4-4dff-ad52-38660d7345fe/discovery/0.log"
Nov 03 11:35:51.153405 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:51.153368    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsn52_86002e1a-3c47-4d7c-ae60-461da50a612c/manager/1.log"
Nov 03 11:35:51.365604 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:51.365568    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb777df86-kcgc4_e3236943-9a1b-4a62-8992-fd40416c20df/kube-auth-proxy/0.log"
Nov 03 11:35:51.444154 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:51.444075    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-74hrv_55911ac4-1644-448a-bf1e-056207cdb3b9/istio-proxy/0.log"
Nov 03 11:35:51.522865 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:51.522830    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-fv94c_56fd579f-dff9-4b92-ac15-00bdf72f86a8/istio-proxy/0.log"
Nov 03 11:35:51.579250 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:51.579216    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hs9mc_7afc9a85-df4f-4401-8656-19750d21605a/istio-proxy/0.log"
Nov 03 11:35:51.642775 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:51.642743    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6fdb79d597-9n4gw_30b7fb6a-bf01-4e56-b60e-dd62207d919c/manager/0.log"
Nov 03 11:35:54.806225 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:54.806194    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kjzdq_93cc76bb-77ed-493e-9207-c5bb8e30bc25/serve-healthcheck-canary/0.log"
Nov 03 11:35:54.900962 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:54.900929    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-xm7qt_b5877c92-09b4-4dff-ad52-38660d7345fe/discovery/0.log"
Nov 03 11:35:54.931159 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:54.931126    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb777df86-kcgc4_e3236943-9a1b-4a62-8992-fd40416c20df/kube-auth-proxy/0.log"
Nov 03 11:35:54.972098 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:54.972071    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-74hrv_55911ac4-1644-448a-bf1e-056207cdb3b9/istio-proxy/0.log"
Nov 03 11:35:55.012111 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:55.012084    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-fv94c_56fd579f-dff9-4b92-ac15-00bdf72f86a8/istio-proxy/0.log"
Nov 03 11:35:55.032623 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:55.032595    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-hs9mc_7afc9a85-df4f-4401-8656-19750d21605a/istio-proxy/0.log"
Nov 03 11:35:55.508724 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:55.508698    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-98krd_59e913f7-bc17-4001-bed8-348466e5966c/kube-rbac-proxy/0.log"
Nov 03 11:35:55.531007 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:55.530985    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-98krd_59e913f7-bc17-4001-bed8-348466e5966c/exporter/0.log"
Nov 03 11:35:55.554541 ip-10-0-134-12 kubenswrapper[5393]: I1103 11:35:55.554515    5393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-98krd_59e913f7-bc17-4001-bed8-348466e5966c/extractor/0.log"